Getting started with Prometheus: the minimum that survives
The minimal monitoring that survives contact with reality: metrics with Prometheus, alert rules that only wake humans, dashboards for two questions.
Monitoring projects die at a fairly specific point: shortly after launch, when the number of named knobs grows, the configured alerts logorrhea, and nobody believes what the panel says anymore. The counter-strategy is small and unspectacular: first the metrics that describe a machine, then the rules that wake a person only when a machine can no longer act on its own. This post describes that minimum with Prometheus, in a shape built to last years.
What the metrics box gives you
A Prometheus with node_exporter on every server immediately answers a small set of questions: is the system up, how high is the load relative to core count, how busy are the disks, how tight is the space on the filesystem, what is memory doing, and how old is the process since the last restart. These questions share two traits: they answer without application knowledge, and their value lies in the trend. A disk that writes 2 percent today, 8 tomorrow and 20 the day after is the signal that arrives early. Anyone who follows five or six of these curves across a small fleet keeps a feel for the state of their systems.
Time series and labels, briefly
Prometheus stores samples as time series and identifies each series by a name plus a set of labels, for example the instance name and the mount point. Once that model is clear, queries stay small: summed across instances, rated over a window, compared against a threshold. The query language is powerful enough to build enormous constructs, but the minimum stays tiny: utilization is the slope of a counter, free space a fraction of a gauge, "is it still alive" a simple existence check.
Alert rules that earn respect
An alert that fires daily gets ignored to death within a month; that is observable human nature. The consequence is harsh selection: alerts only for states that need a human, expressed as an alert that goes pending only after a condition holds for a proof window. That window, written as the for: clause, is an unremarkable detail with a large effect: short spikes do not become alerts, sustained states do. What does not alert goes to dashboards and reports; it stays visible there, but it wakes nobody at three in the morning.
Our standard selection is narrow: machine unreachable, filesystem tight, memory under pressure, service missing. Everything else joins only after these four have spent months in production and their noise floor has proven itself.
Dashboards for two questions
We know the danger of turning dashboards into an art project. The two-question method helps: question one is "is everything up", question two is "where is what is up heading". For question one, a table with the state of machines and services suffices; for question two, the curves of the standard metrics over weeks. Everything else follows demand created by real incidents, not imagination. A dashboard someone opens during a serious moment needs three seconds to an answer, and that requirement sorts visualizations better than any design rule.
Running it without a circus
The operation itself stays small with two components: a Prometheus process and its data store, with a deliberate retention window so the database does not grow forever, and one exporter per machine. Retention and footprint are chosen consciously small: the depth of history is the part almost nobody ever rewinds, while the last few weeks of curves get read daily. If the monitoring server dies, that is the alarm that rings nowhere, so we watch the health of the monitoring itself, most simply through an independent second view that complains when the data gets stale.
What this minimum does not do
Honesty belongs to the plan: this monitoring answers machine questions, not application questions. Whether a shop completes its order flow is invisible to it; that needs indicators from the application itself. Which is exactly why the groundwork comes first: it is the layer every later application measurement stands on, and it prevents the inversion where every department installs its own half-tool. Keep the layers apart and the result is fewer tools and more truth.
Second boundary, equally worth stating: the minimum says nothing about causes. It shows that something is wrong and provides the vantage point from which the cause gets searched. Diagnosis remains human work, and that is good, because automated causal claims are precisely the claims nobody believes anymore.
A closing note on the launch discipline: this minimum deserves a short briefing. The person reading the curves for the first time should know what they will not find there, namely the application perspective. Knowing that, they ask at the right moment for the right tool, and expectations stay calibrated. One sentence in the team wiki fulfills that purpose completely.
If you want monitoring of this kind set up for your servers, or an existing setup reviewed critically, our Linux administration page describes the service, and our post on zero-downtime nginx deployment with systemd shows how deployment and observation play together.
Sources
- Prometheus documentation, Overview, https://prometheus.io/docs/introduction/overview/ (accessed 2026-11-04)
- Prometheus documentation, Alerting rules, https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ (accessed 2026-11-04)
- Grafana documentation, Getting started, https://grafana.com/docs/grafana/latest/getting-started/ (accessed 2026-11-04)