Taming log files on Linux servers

Nobody misses log management while it works. The lesson arrives all at once, usually in one of two flavors: a disk fills up because a service has been writing thousands of lines a minute for months, or an incident review starts and the relevant lines have long since been rotated away. Both are avoidable with about half an hour of configuration work per server. This post covers the two tools that are already running on most Linux servers anyway: the systemd journal and logrotate.

What the journal holds and what it costs

On a current Linux server, journald collects the output and log messages of most services into one structured journal. That is convenient: journalctl filters by time, unit, priority or boot, and journalctl -u nginx.service -f shows the live stream of a service without anyone having to guess the file path first. The cost is disk space. Without limits, the journal grows with the traffic the server sees, and on small systems that collides with the wall sooner or later.

The limit belongs in configuration, not in manual work. Through a drop-in file for journald.conf we set a maximum volume, for example SystemMaxUse=1G, and decide whether the journal survives reboots (Storage=persistent) or exists only until the next one. On servers that matter, persistent is the right call; otherwise the investigation of a crash starts from zero every single time.

That alone is not enough, though. A cap stops unbounded growth, but the question "how long do we need the lines from February?" still needs an owner. That is what vacuuming is for: journalctl --vacuum-time=90d removes everything older than 90 days, and --vacuum-size clamps the journal to a target volume. We treat that as a recurring job or as configuration that enforces the bounds at write time, because a vacuum that only runs when someone remembers is a vacuum that never runs.

logrotate for the classic text files

Alongside the journal live the classic text logs in /var/log: the package manager, mail relays, web servers, our own applications. logrotate is the standard tool for these. It typically runs once a day, works through its rule files, and handles the parts nobody thinks about: compressing and rotating old files, deleting them after the retention period, and telling the affected service to reopen its log file.

That last part is where configurations break in practice. A rotated log is worthless if the daemon keeps its file handle and keeps writing into the renamed file. So every rule file that concerns a service gets either copytruncate or a postrotate script that reloads it. Each has trade-offs: copytruncate can drop lines during the brief copy phase, while a reload is the cleaner path but needs the right signal method per service. We get this wrong once, see the gaps during an incident, and never again.

The second classic trap is retention itself. Distribution defaults are a starting point, not a policy: four weekly logs, one month of syslog. That answers yesterday's debugging question and fails the question of how an incident three months back began. The right number depends on the server, not on the template. A rule of thumb that works: keep service logs with incident potential longer than plain traffic logs, and tie the retention window to a volume someone actually agreed to store.

Forward before the server falls over

On a single server, journal plus logrotate are fine. Once several servers are involved, a second question appears: where do we look when the server itself is the casualty? A machine that takes its last relevant log lines down with it is a painful source. The answer is forwarding: a central collector that receives logs from all servers in parallel, with rsyslog as the proven transport on the sending side.

The often-forgotten piece is the split of responsibilities: the central store gets its own, generous retention, while the local one can stay lean. That gives us enough context for post-incident work without hoarding months of raw data on every host. And we verify the coupling: a relay that silently stops accepting produces no errors on the senders, only unpleasant surprises later.

The checklist for an afternoon

For everyone who wants to work through this once per server, in compact form:

• Configure the journal persistent with a clear SystemMaxUse, then measure the actual volume.
• Fix a vacuum rule that matches the intended retention and let it run automatically.
• Check logrotate rule files for postrotate or copytruncate, ideally walk through one rotation day deliberately and inspect what lands where.
• Define retention periods per server class and write them down, so the next reader knows why it is 90 days.
• With multiple servers: set up forwarding and trace one message all the way to the collector.

What we set by default on client systems

Working across many servers has produced a default we apply without much debate: the journal stays persistent and gets a limit at roughly a tenth of the system disk, with a vacuum rule setting 90 days as the standard retention. Text-based logs follow the same line: weekly rotation with compression, three months kept for service logs, one month for anything that only records traffic. Where a client needs longer windows, they get set per server class, not globally, because a deadline that applies everywhere ends up fitting nowhere.

The second part of the default is monitoring the monitoring: a weekly check that compares the actual volume of the log directories against the expected size costs a few lines of automation and finds the two classic cases early, the ones that otherwise surface at a full disk: a service that suddenly writes tenfold, and a rotation that has quietly stopped working. Both are cheap to find and expensive to miss.

The payoff shows up exactly when it counts: at the next incident, the lines are there, complete, searchable, and the disk is still not full. If you want us to set these baselines up for your fleet or take a second look at your existing log setup, the Linux administration page describes the matching service, and the contact page is the fastest way to reach us.

Sources

• freedesktop.org documentation, journalctl(1), https://www.freedesktop.org/software/systemd/man/journalctl.html (accessed 2026-11-03)
• freedesktop.org documentation, journald.conf(5), https://www.freedesktop.org/software/systemd/man/journald.conf.html (accessed 2026-11-03)
• logrotate project, https://github.com/logrotate/logrotate (accessed 2026-11-03)
• rsyslog documentation, https://www.rsyslog.com/doc/ (accessed 2026-11-03)