Ansible across a large number of hosts: structure beats a script pile
From a script pile to maintained playbooks and roles across roughly 1,000 hosts: structure, update discipline, and the central execution point.
Environment sizes have a way of making problems honest. When the fleet we help operate grew past the point where a folder of hand-made shell scripts could still be described as automation, the script pile had to become a maintained set of playbooks and roles. The fleet counts in the hundreds, with the overall estate reaching roughly 1,000 hosts across teams and locations. This post is the anonymized practitioner report of that change: what structure actually solved, and what a central execution interface changed for daily work.
The script pile, remembered fairly
To be fair to the pile: it worked, for years, and it worked because it was written by people who knew exactly which machines they were touching. The problems arrived with scale and time simultaneously. The same upgrade procedure existed in four dialects, only one of which handled the package manager's new preference file prompting; one box had been excluded by a comment that nobody dared remove; and the question "which hosts did we already fix in the last window?" was answered by memory.
The cost was never the scripts themselves. It was the absence of declarations: no machine-readable statement of desired state, and no execution record.
What structure solved
The move to playbooks and roles replaced procedures with declarations, in three layers. Inventory became the single way to describe the estate, with groups that reflect reality (location, role, environment) instead of convenience. Roles collected the recurring intents: update procedures, hardening steps, service-specific configuration. Playbooks composed those roles into visible, reviewable operations: here is what an update window does, here is what a new web host build does.
Three effects followed almost by themselves. First, review became possible: a proposed change to an update procedure is now a diff someone can read in minutes, and rejection costs a comment rather than a silent fork. Second, drift became visible, because idempotent tasks answer "what would actually change if we ran this" instead of "whatever happens". Third, the fleet's diversity shrank, because maintaining two local variants of one role is more work than unifying them, and the structure makes that work visible every time someone pays it.
Uniform update and compliance checks run over everything, with the same procedure and the same reporting shape, so an update window is measured in what was executed rather than in what somebody believes was executed.
Reporting deserved its own word in that sentence. Every run writes a result the next person can audit, which sounds obvious until one remembers that the script era answered the same question with a terminal scrollback nobody kept. The measurable outcome of the switch was not raw speed. Individual runs take about as long as before; what changed is that they compose. A new machine reaches the estate's baseline through one documented operation instead of inheriting whatever the installing engineer remembered to copy.
The central execution point
The second half of the story is a central execution interface. Running automation from a controller rather than from someone's laptop changes the daily texture of operations. AWX, the upstream project behind the managed variants of such controllers, gives us the unglamorous trio that makes automation trustworthy: traceability (which run executed which revision against which hosts, with logs kept by the platform), permissions (a new colleague can run the update playbook without owning the repository), and scheduled runs (the weekly compliance check exists even when everyone is on vacation).
The cultural shift is quieter than the technical one. With a controller, automation stops being a personal tool and becomes shared infrastructure. That brings discipline to both sides: the playbooks have to deserve being run by others, and the consumers can finally rely on standard outputs. The visible side effect was a quieter calendar: the weekly window runs itself, exceptions arrive as tickets with context attached, and the team's attention moves from the routine to the exceptions.
What we would do in the same order again
If we had to do it again, the sequence stays: inventory first, then roles for the recurring intents, then the controller, and honest exceptions last. The one change we would make earlier is introducing the central interface sooner; it is tempting to see it as a luxury for later, and in practice it is the component that turns the structure into a durable habit rather than a season's enthusiasm. The same applies to the reporting layer: boring, uniform outputs are what make an estate comparable at all.
For the host-level counterpart of this story, the identity stack post describes how directory services and host-side authentication were brought into the same disciplined shape, and the one-year account of rootless Podman operations pairs well if container estates are your concern. The support pages on this site describe how we operate managed estates of this size for customers, and the contact page is the direct way to reach us.