Restore drills: why we regularly replay our backups

A backup that has never been restored is a hypothesis. Everything else, the green checkmark in the job report, the full disk on the target, the monthly e-mail that says success, is about the copy, not about the recovery. This autumn we stopped believing our own reports and ran an honest recovery test on a self-chosen set of machines, from start to finish, including the parts nobody likes to test: DNS records, certificates, and the small state that lives between the lines of a config management run.

This post is the report of that drill, written down while the details were still fresh. It is deliberately a practitioner report about our own servers, not a product comparison.

The rules of the drill

The starting point is the classic 3-2-1 rule: three copies of the data, on two different media, one of them off site. We treat it as necessary but nowhere near sufficient, because it says nothing about the one question that matters: can a given machine be rebuilt to a working state by following a written procedure, in a bounded time?

So the drill had fixed rules. One system, chosen roughly at random, had to be restored fully. Every other system got a spot check: restore one defined dataset, verify one defined service. A stopwatch ran from the moment we decided "restore" to the moment the first real request was answered. Time was measured, not estimated, because an unmeasured recovery time is not a recovery time, it is a hope.

The drill ran on copies, in a quarantine network, never on the production host itself. That last point costs a little setup and buys the certainty that the exercise cannot become the incident.

What worked

The core restore path held up. File-level restores from the current week were cleanly readable, the database dumps from the latest job loaded without errors, and the virtual and physical systems we rebuilt from full images came up at the expected size. That sounds like praise and it is meant as baseline hygiene: if this layer had failed, everything above it would not have mattered.

The pleasant surprise was documentation. The restore notes we had written under a strict rule, "write it for the colleague who does not share your memory", were detailed enough that rebuilding one of the machines was mostly mechanical reading and typing. That rule costs ten minutes when it is written and saves an evening when it is needed.

What was missing

Then came the instructive part. The first full rebuild stopped at DNS records: the zone files were in the backup, but the current serials, some AAAA entries, and one delegated subzone existed only in the DNS panel's own state, which the job never touched. Half an hour later the second gap appeared: the archive we had kept for the old certificate keys could not be opened, because the passphrase for that one container was stored nowhere we could find on a Sunday evening. It resurfaced eventually, in a password note whose reference to the archive was a filename that had changed a year earlier.

The third gap was the subtle one. Two machines came back, answered ping, answered SSH, and then quietly misbehaved: a monitoring agent was still pointed at the pre-migration manager, and one service expected a shared secret that had been rotated after the last full restore was verified. Both were found because the drill forced a real request through the system, not a health check.

None of these are backup failures. They are failures of the space around backups, and the only way to see them is to do the walk on a quiet weekend rather than during an outage.

The field values we use now

For small servers, three figures came out of the stopwatch and two notebooks. Recovery of a single machine, from the decision to the first working request, should be schedulable in half a day; if the procedure cannot plausibly finish in an afternoon, the procedure or the backup scope is wrong. Restore verification has to be scheduled, not occasional; ours now runs as a quarterly drill on one random system plus the spot checks, because six months is comfortably longer than the half-life of "we know how this works". And every item of state that lives outside the backup, be it DNS panel contents, certificate passphrases, or rotateable secrets, needs an entry in the restore document that says where it lives and who may read it.

What changed after the drill

The drill rewrote our own restore document: a dependency list at the top, the out-of-band state items each with an owner and a location, and a hard order of operations from DNS upward. The retention policy got one honest look and one simplification, because keeping more generations than we could meaningfully restore was cost, not safety.

The next drill is already scheduled. A backup we trust is one that has survived several restore runs and a change of season. If you would rather hand this exercise to someone with a stopwatch and a checklist: the support pages on this site include exactly this kind of verification, the blog is where drill notes like this one live, and the contact page is the direct way to talk about it.