Site Reliability Engineer (SRE) interview questionsIncident review — a rate limit pushed everywhere round

Purpose

Makes the incident nobody's fault in the room, so the candidate can criticize the response freely and the interviewer can watch how they do it.

Purpose

Incident response is read here. Separates a candidate who has led incidents from one who has attended them. The first splits the 99 minutes into phases and finds the decisions that cost time; the second summarizes what happened and says communication could have been better.

Signals to score

• Splits the 99 minutes into phases with durations — 29 undetected, 9 to declare, 12 to find the change, 13 waiting to roll back, 19 after the rollback failed, 17 to recover
• Says the 29 undetected minutes cannot be saved in the channel, and sets them aside for the monitoring discussion rather than blaming the on-call engineer
• Names "what changed just before this started?" as the question that should have come first after declaring, and points at the 14:00 push lining up with the 14:02 graph
• Says a recent change is rolled back first and proven guilty afterward, and that waiting 13 minutes for the author's confirmation was a choice
• Says that when the rollback failed at 15:05 the platform team should have been pulled in at once, and counts the 19 minutes that followed
• Weighs pushing the corrected value to all three regions at once against region by region, for a revert to a known-good value
• Asks when the 50 largest customers were told, and says they should have heard before the root cause was known
• Treats the nine minutes spent checking whether the customer was over its limit as reasonable rather than as the main failure
• Distinguishes the incident commander's job from debugging, and asks who was doing which

Follow-up questions

• At 14:40 the incident is declared. What is the first question you want answered?
• The rollback waited for the change's author. Would you have waited?
• The rollback fails at 15:05. What do you do in the next two minutes?
• Region by region took 17 minutes. What would all three at once have risked?
• When did the customers who were being rejected hear from us?

Purpose

SLOs and alerting are read here. Tests whether the candidate can find the blind spot in an objective that looked reasonable, and fix it without creating a noisy one. The separator is whether they see that the failure was concentrated in a few customers and that averaging it into all traffic hid it.

Signals to score

• Names the SLO's exclusion of every 4xx as the reason, and says a 429 the gateway issues by mistake is the service failing, not the client
• Separates rejections of customers over their contracted limit from rejections of customers under it, and says the gateway already knows which is which
• Counts rejections of customers within their contract as failures in the SLI
• Works out that 3% of all requests failing is a burn rate of about 60 against 99.95%, and roughly when an hour-long window would cross a fast-burn threshold
• Sees that the damage sat in fifty customers and that an objective over all traffic dilutes it
• Proposes an objective or an alert for the largest customers as a segment, and shows it would have fired within a minute or two
• Ties a check to the change itself — the rejection ratio before and after a push — and hands it to the rollout
• Rejects an alert on the raw count of 429s, and says what it would do on a night with an abusive client

Follow-up questions

• The SLO excludes all 4xx. Who decided a 429 is always the client's fault?
• The gateway knows each customer's contracted limit. What could it count that it does not count today?
• 3% of all requests failing against 99.95%. How fast is the budget going, and when does a one-hour window notice?
• The failures were in fifty customers. What does averaging them into 120,000 requests a minute do?
• Would an alert on the number of 429s have helped?

Purpose

Safe automation is read here. Tests automation that changes a fleet safely — progressive exposure, a gate that reads what the change can break, and a way back that is exercised rather than assumed. The separator is whether the bake reads client-facing signals and whether rollback is designed as re-applying something known to work.

Signals to score

• Stages by blast radius — a few instances in one region, then that region, then the rest — rather than a percentage of the whole fleet at once
• Makes each stage wait on what the change could break — rejection ratios, errors and latency seen by clients — not on process health
• Compares instances that took the change with ones that did not over the same minutes, rather than against a fixed threshold alone
• Halts and rolls back automatically when a stage breaches, and says what a human is told
• Makes rollback re-apply the stored previous artifact exactly, without re-rendering or re-validating it against a newer schema
• Keeps both schema versions readable during a migration so that going back never crosses a version the fleet cannot read
• Adds a check on the size of a change — a limit cut by 91% gets a before-and-after in per-minute terms and a second confirmation
• Exercises rollback routinely — on a canary with every push, or as a drill — so the first real use is not the first use
• Keeps a documented way to push a known-good value fast, with the stages skipped on purpose, and says who may use it

Follow-up questions

• Your first stage waits on something. What is it, and would it have seen 38% of fifty customers rejected?
• The rollback failed because it re-validated. What is a rollback, exactly, in your design?
• A change reduces a limit by 91%. Should the tool treat that like any other change?
• When did your rollback last work, and how do you know?
• You have made rollouts slower. What did that cost, and who pays it?

Purpose

Incident response is read here, in the part that happens after recovery: whether the candidate can write a review that changes the system rather than one that assigns a cause to a person, and whether the actions it produces are ones someone will actually finish.

Signals to score

• Replaces "human error" with the conditions that let a typo reach customers: validation, review, rollout, detection and rollback, each named
• Restates the impact in the terms of the people it hit — 38% of requests from the fifty largest customers for 99 minutes — not only 3% of all traffic
• Keeps the rollback item and says why it is the only one that survives as written
• Drops "be more careful" and "add more monitoring" as unverifiable, and replaces them with specific changes, each with an owner and a date
• Turns "retrain the team" into a change to the tool or its output, and says what the training was standing in for
• Corrects "fast response once declared", since recovery took another 61 minutes
• Adds the missing 29 minutes to the timeline, with what each system showed during them
• Keeps the engineer out of the root cause, and asks what made 3 look right to them and to their reviewer

Follow-up questions

• The root cause names a person. What would you write instead?
• "About 3% of requests." Who would read that and feel misled?
• Which action item would still be open in six months, and why?
• What went well, honestly?
• What would you ask the engineer who typed 3?