LVM snapshots: the rollback before the update

Not every change on a server is risky, but the ones that are share one property: they only reveal themselves as mistakes after the fact. A distribution upgrade, a kernel update with new drivers, a configuration package that rewrites half the services. For exactly those cases there is a safety net that is already installed on most Linux servers and only needs to be used: LVM snapshots.

What a snapshot technically does

An LVM snapshot is not a backup; it is a block-level point-in-time view with copy-on-write: from the snapshot moment onward, writes go into a separate area while the original blocks stay frozen in the snapshot layer. Two consequences follow, and both matter.

First, a snapshot is cheap to create but not free: the more the original changes after the snapshot, the more delta blocks land in the snapshot area. So every snapshot needs a size budget and ideally an autoextend rule; otherwise it fills up and becomes unusable at exactly the moment it is needed. Common practice is a tenth to a quarter of the original volume, depending on the expected write share, plus watching the usage while the change runs.

Second, the point-in-time view is sharp at the block level, not at the application level: a database service in the middle of a transaction leaves behind a state the database has to reconcile first when played back. For file shares and system volumes this is irrelevant; for databases, a snapshot is part of a plan, not the whole plan.

The standard procedure before a change

The pattern that works in daily operations is small enough for four steps:

• Create a snapshot of the logical volume in question before the change, named with date and purpose.
• Run the change: apply the update, reconfigure, switch the kernel.
• If everything holds, delete the snapshot after a defined window, for example two uneventful days.
• If it does not hold, roll back instead of repairing.

That last step is the one that makes the difference. Instead of trying to un-wind a failed update step by step under time pressure, we stop the machine, let the snapshot layer merge back over the original, boot again, and a few minutes later we are standing in the state before the change. That costs a short planned downtime, not a repair hour with an uncertain outcome.

One detail deserves its own respect: the merge typically requires a reboot, because the root filesystem cannot be rebuilt underneath a running system. We rehearse the procedure once on a test machine or a recovery box before doing it live for the first time. That is what a quiet afternoon is for.

Thin snapshots as the modern variant

Besides classic snapshots, current LVM versions offer thin volumes with snapshots that behave differently: the snapshot layer starts almost empty and grows with the blocks actually changed. For rollback purposes before updates they are the more convenient variant, because several snapshots can be kept in parallel, one before the update and one before the configuration change that follows. The price is a bit of administrative discipline: thin pools want to be watched, and the same rule holds here, that a full pool surprises at the worst time.

What the snapshot does not replace

The limits, stated honestly: an LVM snapshot lives on the same hardware as the original, and a disk failure takes both. A snapshot is also not a backup of data spread across systems, and it replaces neither the tested restore from a backup nor the application consistency a database secures with its own tooling. The sensible order is therefore: snapshots for the operations-adjacent return, backups for the event that takes the server as a whole, and the database with its own consistency tools.

Which changes deserve such a safety net is decided by the triage of the own patch queue; the post CVE triage: how we set priorities shows how we sort it.

Which changes deserve such a safety net is decided by the triage of the own patch queue; the post CVE triage: how we set priorities shows how we sort it.

Which changes deserve such a safety net is decided by the triage of the own patch queue; the post CVE triage: how we set priorities shows how we sort it.

Which changes deserve such a safety net is decided by the triage of the own patch queue; the post CVE triage: how we set priorities shows how we sort it.

How the practice fits into operations

For the safety net to stay more than theory, it belongs inside the change process itself. In our practice that reads: no update window on an LVM system starts without the snapshot step, and the sequence stands in the change entry, not only in a manual. Whoever runs the snapshot as its own item in the procedure does not feel it as overhead but as one minute of preparation that relaxes the rest of the operation.

The second practical point is the deadline: a snapshot lying around for weeks is no longer a rollback, it is waste with costs. So the procedure carries an end date on which the snapshot gets deleted, whatever the outcome was. The exception is the case where the change is still under observation; even then a deadline runs, only a longer one. Quick cleanup keeps the snapshot list short enough to read in seconds during an incident, and that is exactly its purpose: not a collection of save points, but readiness for one question.

One detail from restore practice deserves its own paragraph: the return over the snapshot is the moment that shows whether the documentation is true. Whoever reconciles the change note against the actual state after the merge finds the small deviations between plan and execution that otherwise stay undiscoverable: the one package list that looked different, the one service named differently in the test. Those reconciliations are the reason the return is possible without nasty surprises, and with today's tooling they take minutes.

If you want such a safety net established across your fleet before the next big update, the DevOps and backup service page describes the matching service, and the contact page is the fastest way to reach us.

Sources

• LVM2 project resources and documentation, https://sourceware.org/lvm2/ (accessed 2026-11-03)
• lvcreate(8) manual page, https://man.archlinux.org/man/lvcreate.8.en (accessed 2026-11-03)
• lvmthin(7) manual page, https://man.archlinux.org/man/lvmthin.7.en (accessed 2026-11-03)