Central sudo logging: what an audit server for many hosts taught us

Collecting audit events from an estate of roughly 1,000 hosts centrally sounds like a product decision, but most of its value turned out to be a handful of rules about what to ship, where to protect it, and what to actually read. This post is the anonymized practitioner report of running sudo session capture and audit event streams into one central point: why local logs alone fall short, what the pipeline looks like without heavy per-host overhead, and which messages earn real reading time.

Why local logs alone fall short

A host's own audit log has three structural weaknesses that no local configuration fixes. It dies with the host: an attacker who reaches the machine can erase the record of reaching it, and a rebuilt host can lose the forensic context of its own history. It is uncorrelated: the interesting sequences in an incident span several machines, and matching them from hundreds of separate local stores by hand is hours of work at exactly the moment hours matter least. And it is unpoliced: nobody notices that one host's log shipper has been silently failing since winter.

Centralizing fixes all three by moving the authoritative copy off the machines being audited, and that move determines the rest of the design.

The pipeline without subscriber overhead

The shape that survived in practice is deliberately unfashionable: each host ships the defined streams, sudo session events and the audit messages worth shipping, over a simple, authenticated, buffered transport to one receiving tier, which stores first and processes later. No per-host subscriber logic, no enrichment on the wire. The receivers store the raw stream immutably, and interpretation happens downstream, on a copy.

Three details carry most of the reliability. Buffering on the hosts means a network partition or a receiver maintenance window delays delivery instead of losing it. Authentication between hosts and receivers is a certificate story, which doubles as the inventory: if a host cannot authenticate, the gap appears as a monitoring event, because "is the estate actually shipping?" is a monitored question, not an assumption. And the receiving tier's write path is append-only with access separate from the audited machines' operators, so the audit trail's protection does not depend on the good behavior of the systems it audits. Rehearsal closes that loop: twice a year, one defined incident question gets pulled through the store end to end, from raw stream to written answer, inside a fixed time budget.

On the host side the overhead question is answered by scope discipline: ship the defined events, not everything. Full audit streams from a large estate produce a firehose that turns the central store itself into an operational problem. The sudo capture rule that proved itself: sessions and command lines for administrative access, with the usual exclusion debates settled by policy rather than by per-host improvisation.

Which messages actually get read

The honest lesson: most of the collected volume is never read, and pretending otherwise wastes the budget for the parts that always get read. The streams that earned permanent reading time in our operation: administrative logins from unexpected sources or hours, sudo usage on hosts where the owner teams changed recently, audit records about changes to the shipping and logging machinery itself, and the completeness metrics that answer "did everything ship today".

Everything else gets aggregates and alarms instead of eyeballs. The reading pattern follows the questions an incident actually asks: who, from where, changing what, since when. Designing the retention for those questions beats retaining everything and searching nothing.

Retention deserves the same honesty. A year of shipped sessions taught the query patterns faster than any design meeting; the store now keeps the security-relevant categories at full detail for exactly the depth our own incident reviews actually reached back to, then rolls them into the aggregate view. The completeness metrics stay hot the whole time, because a gap in the trail is a security fact in its own right.

What it taught us about audit as an operation

An audit pipeline is an operated service, with patching, monitoring, and its own restore story, rather than a checkbox. The events worth the most are the ones about the pipeline itself, because a quietly degraded audit trail produces the worst version of a surprise. And the cultural effect compounds: once teams know privilege use is centrally visible, the asking-permission behavior improves before any rule enforcement does, which is the cheapest security effect we have ever observed.

For the estate context this operated in, the Ansible structure post describes how the host fleet got its consistent shape, the custom roles post shows the discipline behind the shipping and exclusion rules, and the identity stack post covers the authentication layer the audit events ride on. If a central view over privilege use and audit events is something your environment lacks, the support pages describe how we run such pipelines for customers.