These questions test how you behave when production breaks: staying calm, leading the response, owning your mistakes, preventing the next one and keeping on-call humane.
Q32Describe a time you prevented an incident before it happened.
🧠 Psychology: what they're really testing
They're probing proactivity: do you see risk before it bites? Prevention is invisible work, so they want evidence, not "I keep an eye on things". Red flags: vague claims with no numbers. A senior answer forecasts, tests the assumption safely, measures the headroom and makes the non-event visible.
🧭 Philosophy: the principle behind a strong answer
The best incident is the one that never happens, but prevention must be proven, not assumed. Game days and load tests turn assumptions into evidence. As the SRE saying goes, hope is not a strategy.
SSituation: The Australian end of financial year (30 June) brings a surge in tax-related transfers, super contributions and statement downloads. Our platform had moved to the cloud that year, so last year's capacity numbers no longer applied.
TTask: I owned the platform's resilience plan and had to prove we could handle the expected peak.
AAction:
- I built a forecast with finance and product: about 3.2 times a normal day at the peak hour.
- I ran a game day in a prod-like environment, replaying shaped traffic at 1x, 2x and 3.5x.
- At about 70% of the expected peak, autoscaling stopped. We'd hit a cloud vCPU quota ceiling and a database connection limit that nobody had re-checked after the migration.
- I raised quota increases with the provider, tuned connection pooling and pre-scaled the day before. I considered freezing statement downloads, but it wasn't needed once the limits were fixed.
- I re-ran the game day to 3.5x, passed, and wrote a peak-day runbook with a named decision-maker.
RResult: Peak day reached about 3.4x normal with zero customer impact, and p95 latency stayed within SLO. I shared it with leadership as "the near-miss we created on purpose". Game days became a quarterly practice before every known peak.
R+Reflection: Non-events are invisible, so I quantify them: "we would have failed at 70% of peak". I'd also automate quota checks after every migration.
Power words to land: forecast · game day · headroom · quota ceiling · evidence, not hope · peak-day runbook
The most useful postmortem I ever wrote started with: "I approved this." 🧯
🔁 3 instant retries, no jitter, felt safe. It turned a 4-minute blip into a retry storm
🙋 Owning it first made it safe for others to own theirs
📏 The fix became a standard: backoff + jitter, circuit breaker, retry budget
🧪 And a chaos test, so we prove it, not hope it
Blameless doesn't mean nobody is accountable. It means leaders go first. What's the best postmortem habit you've seen?
#IncidentManagement #SRE #Leadership #Resilience #EngineeringManagement