Site Reliability Engineer (SRE) interview questionsSLO review — 212 pages and a missed outage round

Purpose

Removes definitions as a place to spend the hour, and says the round ends in decisions rather than a discussion.

Purpose

SLOs and alerting are read here. Separates a candidate who has run a service against an SLO from one who has read about them. The first starts from the outage the SLI missed and the target the service never met; the second starts from the nines.

Signals to score

• Notices that the target has been missed every month by the draft's own formula and says a target nobody acts on is decoration, not an objective
• Uses 19 August to show that the SLI cannot see a failure returned as HTTP 200, and says that measurement is the first thing to fix
• Measures the outcome a customer cares about — an order accepted — at the service, from what the application knows, rather than status codes at the load balancer
• Decides explicitly which failures count against the service and which are the client's, such as a rejected malformed order
• Sets a target from what the service achieves and what customers need, not from the contract or from habit, and says the number
• Chooses a rolling window over a calendar month, and says what a calendar reset lets a team forget
• Works out what 19 August would have cost had it been counted, against the old target and the new one
• Adds a way to see failures the service cannot report about itself, such as a synthetic order, and says what it covers
• Names who owns the SLO and what is supposed to happen when it is missed

Follow-up questions

• June, July and August all missed 99.99%. What happened as a result?
• On 19 August, what did your SLI say about 13:05 to 13:52?
• You are measuring at the service now. What failure can the service not see about itself?
• What would 19 August have taken out of a month's budget at 99.99%? At 99.9%?
• Why rolling 30 days and not the calendar month?

Purpose

SLOs and alerting are read here. Tests alerting on symptoms against alerting on causes, and whether the candidate can turn an SLO into a page with real thresholds rather than a principle. The separator is whether the replacement alert comes with numbers, and whether they check it against both 19 August and a quiet night.

Signals to score

• Takes `HighCPU` and `PodRestarts` off the pager using the pack's own count — 135 pages, none acted on
• Says why they fail as pages: they report causes that may hurt nobody, not symptoms a customer feels
• Explains the night-time `Error5xxRatio` pages from the traffic figures: one failed request out of about 40 in a minute is 2.5%
• Replaces the ratio threshold with alerts on how fast the error budget is being spent, each over a long and a short window together
• Gives the fast-burn page its numbers — a burn rate, both windows, and the error ratio that rate means at the new target
• Shows roughly when that page would have fired on 19 August, now that the SLI can see the failures
• Adds a slower-burn alert for problems too small to trip the fast one, and says whether it pages or files a ticket
• Keeps `OrderSubmitLatencyP99` and says what makes it the one alert in the file that works
• Pages on `DiskUsage` only when the disk will fill before anyone could act in working hours, and files a ticket otherwise
• Applies the same test to the nine rules not listed, rather than leaving them alone

Follow-up questions

• `HighCPU` paged 83 times and nobody did anything. Is there any reason to keep it?
• At 03:00 a single request fails, out of the 40 in that minute. What does your new alert do?
• Give me the numbers: the burn rate, the long window, the short window, and the error ratio that means at 99.9%.
• On 19 August, at about what time does your page fire?
• Three of the four disk pages were real. Page or ticket?

Purpose

Changes the interviewer's role explicitly, so the pressure that follows is heard as the product side's and not as a trick.

Purpose

Reliability trade-offs are read here. Tests reliability as a trade-off negotiated with the people who own the product, rather than a rule enforced on them. The separator is whether the candidate can hold the line on measurement without making the release decision theirs alone, and whether what gets written down would settle the next argument before it starts.

Signals to score

• Separates the contract from the internal objective, and asks what the contract measures and what it pays when missed
• Says that measured honestly the service has not been delivering 99.99%, so the contract is a business risk to raise now, not a number to copy into the SLO
• Does not reach for an automatic release freeze, or for a waiver, and says who should make the call
• Proposes a written error budget policy agreed in advance: what happens when the budget is spent, what is exempt, and who can grant an exception
• Offers a way to ship that risks less — behind a flag, to a small share of traffic first, with the fast-burn alert watching and a rollback that has been tried
• Treats a change that reduces risk differently from one that adds it when the budget is spent
• Uses the numbers from earlier in the round rather than arguing in general
• Says what reliability owes product in return: a measurement both sides trust, and a policy applied the same way every time

Follow-up questions

• The contract says 99.99%. What has the service actually been delivering?
• Is "the budget is spent, so nothing ships" your decision to make?
• What would let the redesign go out next week without betting the month on it?
• Who signs an exception, and where is it written down?
• The redesign also fixes how payment errors are handled — the 19 August failure. Does that change your answer?