Seeing disk failures before they hit
How disks announce their deaths: SMART attributes that predict failure, scheduled self-tests, mdraid event handling, and a spare in the drawer.
Disks rarely die quietly. They announce themselves, often for weeks, in values nobody reads. The difference between a survivable Tuesday and a lost weekend is therefore rarely hardware budget but attention structure: whoever reads the disks' signals regularly buys time to act; whoever only watches the RAID console learns of the first fault when the second is near. This post is the line we draw on servers with owned hardware.
The attributes that actually predict
SMART reports dozens of values, but three carry most of the predictive weight: Reallocated Sector Count, the tally of already replaced bad blocks; Current Pending Sector Count, the blocks still waiting for a verdict; and the offline uncorrectable values, errors that could no longer be corrected. Depending on the model, seek-error relatives join the core. The key insight is the inertia of these values: a single bump in one night is a reason for attention, not replacement; a rising trend across two weeks is an appointment.
Beyond the attributes we schedule self-tests: short tests in unloaded windows and occasionally an extended test overnight. They surface faults that normal operation touches late, and their results are logged and monitored like attributes.
mdraid: what the array itself sees
On software RAID systems there is a second data source that gets underestimated: the md layer itself. It knows the states of its members, reports when one drops from the array, and, more importantly, when spares enter or fail to. Equipped with mdadm --monitor running continuously, these events move into mail and log. The unglamorous part of the routine belongs with it: a scheduled scrub that reads the array through once a month and makes silent errors visible. On arrays that run correctly, that is the one chance to learn how consistent the disks are with each other.
Something we miss every time a client arrives without it: the device state after replacement. A disk comes out, the new one goes in, the array rebuilds, and nobody ever fully verified the new disk against the data. A scrub after the rebuild closes exactly that gap and therefore belongs to the replacement sequence itself.
The spare belongs to the plan
The third component is material: a prepared spare disk in the drawer, same series or compatible in sizing, bought in advance. The reasoning is plain: the interval between failure and restoration is decided by procurement, not by engineering. A spare standing ready pulls the restoration down to hours. Together with the question of replacement costs, the spare belongs to disk policy, not to budget trimming.
Teams running their own hardware arm themselves this way against the problem that our backup post covers under the banner "RAID is not a backup": the array raises availability, but a controller fault or a power event can cost the consistency that no RAID container reproduces. The backup cycle is the layer that catches those cases.
What we set by default
On the servers we manage the baseline looks like this: SMART values land on a dashboard, the three reliable attributes are exported as metrics, the daily review of changes runs automated, scrubs run monthly, mdadm --monitor runs permanently with a mail address, and the spare-drawer inventory is part of the annual plan. All together it is half an hour per server, set up once, and it shortens the distance between "hopefully we will notice" and "we know".
Learning the signals before they hurt
The underrated part of disk work is learning the normal values. Every machine has its noise: a model with its characteristic self-test rhythm, a controller with its timing signature, a chassis with its thermal habits. Whoever logs attentively for one week knows afterward which behavior is quietly normal on that box, and then sees deviations earlier than any generic threshold tool would. That individual baseline is why a freshly set up monitoring should spend its first weeks learning to read.
Second lesson: the interplay of attributes. A slight rise in the reallocation counter plus rising queue time is a different story than the same rise without wait time. The combination is the information; the single value is a boundary condition.
A second aspect belongs to this attention: storing the values themselves. SMART numbers that exist only in the session terminal are forgotten by tomorrow; they belong in a history. The history is what stock dashboards do not deliver: long-term experience of how fast the disks of this fleet age, and with it the ability to move replacement into a planned window instead of the night. The effort for that bridge is one file and one script, and it usually pays for itself at the second incident.
If you want your servers' disks checked against this line, or your spare inventory organized, the DevOps and backup service page is the right entry.
Sources
- smartmontools project, https://www.smartmontools.org/ (accessed 2026-11-04)
- manpages.debian.org, smartctl(8), https://manpages.debian.org/bookworm/smartmontools/smartctl.8.en.html (accessed 2026-11-04)
- manpages.debian.org, mdadm(8), https://manpages.debian.org/bookworm/mdadm/mdadm.8.en.html (accessed 2026-11-04)