Observability — Logs, Metrics, and Traces in Practice

When the phone rings at 3 a.m., can you answer without logging into production and looking around? Starting from zero, this piece builds up structured logging, metric cardinality, trace context propagation, and SLOs with error budgets — then lines them all up as a single motion: notice, narrow, confirm.

Guessing the Inside from the Outside

Your phone rings in the middle of the night. "Payments are failing." You are not sitting in front of the server, and you could not say where that server physically is. All you have is whatever the system chose to emit about itself while it was running.

It is a lot like being a doctor. You don't open the patient up to look inside — you infer what is happening internally from temperature, blood pressure, a blood panel, and what the patient tells you. Observability is originally a term from control theory, meaning the property that internal state can be inferred from externally visible outputs alone. In software it translates cleanly: can you get all the way to the cause using only the records that already exist, without going back into production to instrument something new?

The difference from the closely related idea of monitoring is who writes the question, and when. Monitoring is a machine for answering questions you decided on in advance — "page me when CPU goes above 80%" — where a human wrote the question first. Observability is the property that lets you answer a question you never anticipated. "Android users in Japan are slow, but only on retries of the payment API" is not a slice anyone thinks of before the incident.

Which is why observability is not the name of a tool. It is a property of your application's design. An app that emits nothing will show you nothing, no matter how expensive the platform you run it on.

Three Signals

• Logs — one record per event. "12:03:41, payment failed for user A."
• Metrics — numbers over time. "Seven payment failures in the 12:03 minute."
• Traces — the path one request took, and how long it spent in each segment.

Think of shopping. A log is a receipt: complete down to the line item, but you accumulate an enormous stack of them. A metric is the chart in your budgeting app: light enough to keep years of it, but it never tells you why. A trace is a parcel tracking number: which depot this one package sat in, and for how many minutes.

The reason you want all three is that their costs grow in different shapes. Metrics are aggregated before they are shipped, so if the number of series stays the same, a hundredfold increase in requests barely changes what you store. Logs scale directly with the number of events. Traces scale with requests × segments per request, so left alone they swell at exactly the rate of your production traffic.

The three connect into a single line of motion: metrics tell you something is wrong, traces narrow down where, logs confirm what. Everything that follows is reinforcement of that order.

Metrics Multiply

The atomic unit of metrics is the time series: one per distinct combination of name plus labels. Four types cover almost everything. A counter only ever increases — the value itself is meaningless, the slope is what you read. A gauge goes up and down and represents a current value (queue depth, memory in use). A histogram counts values into buckets to describe a distribution; latency belongs here. A summary computes quantiles on the client before shipping them.

The question that matters is how many series you end up with.

$N$ is the number of series, $k$ is how many labels you attached, and $c_i$ is how many distinct values the $i$-th label can take. Stated in words: take the number of possible values for each label and multiply them all together, and that is how many lines are quietly created behind your one chart. Attaching another label does not add a row — it copies every combination you already have, once for each value the new label can take. The whole content of that formula is one word: multiplication, not addition. Five methods × six statuses × forty endpoints × twenty pods produces 24,000 series from a single metric name. Add user_id and the entire thing gets multiplied again by your user count.

Averages Lie

"Average latency is 200ms" tells you almost nothing. If 990 of 1,000 requests take 50ms and the remaining ten take fifteen seconds, the average still lands near 200ms. The people who are angry are the ten.

So you look at the distribution instead. A p99 of 800ms means "one request in a hundred takes 800ms or more." And that number interacts badly with the fact that one user-facing operation usually fans out into many calls underneath.

$$P = 1 - (1-p)^{n}$$

$P$ is the chance that an operation hits at least one slow call somewhere, $p$ is the chance any single call is slow, and $n$ is how many calls one operation makes. Put in words, it says: work out the chance of getting through every single call without a slow one, then count all the remaining cases as "hit at least one." Assuming independence, $p = 0.01$ and $n = 100$ gives about 63%. "One in a hundred" becomes "two people in three" once you call it a hundred times. That multiplication is the entire reason tail latency is treated as an emergency and averages are not.