facts before stories

(this images are for illustrative purposes only ;) )

_

compare these two incident updates

version A

I believe service A is causing the incident

version B

I don’t know the root cause yet. the strongest signal points at service A because it shows up in most slow traces. I’m validating that now

A sounds confident. B is actually useful because it tells everyone what’s known, what’s inferred, and what’s still open

that difference is basically the whole post

most incidents drag on not because engineers lack skill but because hypotheses quietly turn into facts through repetition and urgency

the real skill in incident communication isn’t explaining what you think happened. it’s making brutally clear what you know, what you suspect, and what you don’t

how hypotheses become facts

every incident starts with uncertainty. a slow api, a cpu spike, a customer complaining about latency. your brain immediately writes a story. “must be yesterday’s deploy.” “probably the cache.”

fine, those are hypotheses

the investigation starts drifting the moment a hypothesis is communicated as a conclusion. “I suspect the cache” becomes “the cache is causing the latency”

nothing changed technically. only the wording

but once a hypothesis is treated as fact, every step after it is biased toward proving it instead of testing it. engineers unconsciously ignore evidence pointing elsewhere

teams also trust confident speakers. drop in some commit hashes, deployment timestamps, and an architecture diagram, and the hypothesis feels complete

people stop asking “what evidence supports this?” and start asking “how do we fix it?”

a detailed explanation is not evidence. a convincing narrative is still just a narrative until data confirms it

time correlation is the most common trap. latency spiked at 2 pm. a deploy went out at 1:45. it’s tempting to blame the deploy

but maybe database stats changed. maybe a scheduled job kicked off. maybe traffic doubled

correlation is evidence, not proof

every time you catch yourself saying “this happened after X,” ask the follow-up question

what shows X actually caused it?

separate facts from interpretation

one habit fixed more incident discussions on my teams than anything else

split observations from interpretations in writing

customers report slow search

a bad update

search latency is caused by redis

a good one

• observation
  search latency went from 300 ms to 9 seconds after 14:00 UTC

• evidence
  database query duration increased. redis response time stayed flat. cpu doubled. slow traces spend most of their time in database execution

• hypothesis
  the database is the primary bottleneck. I haven’t ruled out application-level contention

• next step
  compare execution plans before and after 14:00 and profile the slowest queries

the facts stay useful even if the hypothesis turns out to be wrong

the reader also knows exactly what’s measured versus what’s guessed

the same discipline applies at the sentence level

“the api feels slower” is intuition

“p95 went from 800 ms to 12 seconds” is a measurement

intuition tells you where to look. metrics tell everyone what to do

precision beats absolutes too

“we fixed it” sounds like someone who really wants the incident to be over, close the case, go to sleep.
“we shipped a change that cut latency by 70% and we’re watching whether it holds” is credible

I’ll take credibility over certainty every time

ask the falsifying question

the quality of an incident channel is set by the questions in it. not “who broke this” but “what changed”. not “is this the root cause” but the single best question I know:

what evidence would convince me this hypothesis is wrong?

good hypotheses are testable. bad hypotheses become beliefs. and once a hypothesis becomes a belief, the discussion turns into a debate someone has to win. wrong game. reality doesn’t care which hypothesis sounded better. the goal is never being right, it’s finding reality fast. every disagreement should end with a measurement, an experiment, or a trace. never with a louder opinion

this is also why changing your mind when new evidence appears isn’t weakness

it’s the job.

---

;