Our routine for diagnosing slow servers

There is a panic question that arrives at half past one: the server is slow, the demo starts in forty minutes, and three people are typing on the same console. Such moments do not reward intuition, they reward an order. This post is our fixed diagnostic line, layer by layer, from system load through disk to network, and with the endpoint where diagnosis turns into a capacity decision.

Step one: read the load before pulling tools

Before any measurement tool starts, the load gets read: what is the load average, how does it split between user, system and wait time, how many runnable processes stand against the core count? That distinction is the hinge of the whole diagnosis. High load dominated by wait points toward disk or network; high load with heavy user time is a computation problem; high load with many structurally sleeping processes is a scheduling problem. Five minutes of reading tells us more than an hour of blind tool operation, and the next layer already has a hypothesis.

Ask the disk

The second layer is the disk, because it is the most frequent author of the subjective "everything is slow". iostat with a short observation interval shows utilization, wait time and throughput per device. What matters is not the peak but the spread: a disk running pinned at its limit while its queues spike periodically points onward, to whatever hangs there, such as a backup window that slid into business traffic. Virtualized environments add a twist: the guest sees only a slice, and the true story plays on the storage backend. Then it pays to look at neighboring instances before digging deeper into the local system.

The difference between "the disk is working" and "the disk is braking" sits in the wait time as the processes experience it, not in throughput. A fat batch system can run at 90 percent utilization while remaining invisible to the applications, while a slow consumer device blocks everything at 20 percent.

Network and memory, queried sharply

The network layer starts with the simple question: does the slowness feel local or remote? ss shows the sockets and their states, and listen queues growing long are the mark of a receiver that cannot keep up. Latency and packet loss to the next hops separate a server problem from a link problem. The memory layer needs less elegance than discipline: vmstat separates runtime tempo from swapping behavior cleanly; when pages continuously flow out and back in, the message is unambiguous, however friendly the cache numbers look. The kernel documents its own interpretation of the load figures, and reading that note once, before the hot moment arrives, is worth it.

The report after the incident

The last piece of the routine costs twenty minutes and pays the longest: a short report. Five questions suffice: who noticed the feeling first, what did the measurements show, which curve would have betrayed it earlier, which measure was taken, what stays under observation? The report is deliberately not a study but half a page that shortcuts the next person's start. After three such reports something appears that no single meeting produces: a catalog of the real disturbance patterns of the own fleet, with the curves and the solutions beside them.

That catalog also shows the payoff of the structure: nearly every entry belongs to one of the four layers this post starts from, so the diagnostic line calibrates itself to the fleet it serves.

One more remark concerns the surroundings in which diagnosis happens: consistency. Whoever pulls a measurement series while colleagues ship deployments in parallel measures mixed signals. The elegant answer is not an elaborate toolchain but a short agreement: change activity pauses during the observation unless it is part of the question. The cost of that agreement is one message in the team channel; its benefit is that the second series can confirm or refute the first.

The transition to capacity

Diagnosis has a natural endpoint: the moment when the answer is no longer a setting but a purchase. The rule reads: when the same bottleneck appears in three measurement series at the same spot, the load is not temporary, and the configuration avenues are exhausted, capacity gets planned, with numbers in hand. That discipline is the counterweight to the permanent optimization that tries to eliminate every boundary with configuration. Not all growth is a fault; some findings are an order.

The feedback loop counts too: after the incident, what was observed gets recorded, which curve would have shown it earlier, and the answer feeds the rules of monitoring. That is exactly what our monitoring groundwork in the Prometheus post exists for, and the disk layer connects with our SMART monitoring post. Feeding both removes the surprise from the next incident. For the structural side of running your servers, the Linux administration page is the entry point.

Sources

• sysstat project documentation, https://github.com/sysstat/sysstat (accessed 2026-11-04)
• man7.org, ss(8), https://man7.org/linux/man-pages/man8/ss.8.html (accessed 2026-11-04)
• man7.org, vmstat(8), https://man7.org/linux/man-pages/man8/vmstat.8.html (accessed 2026-11-04)
• Kernel documentation on load average via proc(5), https://man7.org/linux/man-pages/man5/proc.5.html (accessed 2026-11-04)