DevOps Engineer interview questionsDesign review — a release path where rollback is one commit round

Purpose

Makes this a review of a colleague's proposal whose goal is legitimate, so criticism has to be specific to be useful, and a candidate cannot score by rejecting the whole thing.

Purpose

The load-bearing question. Each proposal sounds reasonable on its own; together they make a canary that migrates the shared database for everyone, a gate that cannot see the damage, and a rollback that makes it worse. The read is whether the candidate puts the proposals together rather than reviewing them one at a time.

Signals to score

• Sees that the canary's first pod renames the column for the whole database while 95% of traffic is still served by code that uses the old name
• Sees that the gate cannot catch it: the canary is the one version that works, so it passes the absolute thresholds and would look better than a broken baseline in any comparison
• Sees that an automatic rollback of the canary at that point removes the only pods that match the schema
• Says reverting the digest puts old code on the new schema, so rollback is one commit only for releases the previous version can run against
• Replaces the rename with expand and contract across releases: add the new column, write both, backfill, read the new, stop writing the old, drop it later
• Moves migrations out of startup into one step that runs before the rollout, and requires every migration to work with the version already running
• Names what happens when several pods start together and each tries to migrate, unless the migration tool takes a lock
• Says the old column can be dropped only once no release that reads it can be rolled back to
• Does not rely on down migrations, because one that drops what a release added drops what was written to it
• Proposes a CI check that flags a destructive schema change arriving in the same release as the code that needs it

Follow-up questions

• The first canary pod starts. What happens to the other 95% of requests?
• What does the canary gate see during those ten minutes?
• The canary is rolled back automatically. What happens next?
• How many releases does a safe rename take, and what is in each?
• When can the old column finally go?

Purpose

Arithmetic separates this one. Candidates who have tuned a canary work out how many requests it sees and how many of those touch the path at risk; candidates who have read about canaries argue about thresholds.

Signals to score

• Works out the sample: 5% of 12 requests a second for 600 seconds is about 360 requests
• Works out that refunds, at 0.5% of requests, are about two of those 360, so breaking every refund adds about half a point to the canary's error rate
• Sees that on top of the normal 0.3% that comes to about 0.8%, under the 1% threshold, so the release is promoted
• Says an absolute threshold ignores the baseline, and compares the canary with the baseline over the same minutes instead
• Knows that telling 0.3% from 0.8% reliably takes thousands of requests, and says what that means for weight or duration at this traffic
• Adds checks for the rare paths that matter — a synthetic refund against the canary, or refund success as a signal of its own — instead of relying on the aggregate
• Steps the weight up in stages rather than jumping from 5% to all traffic
• Uses a feature flag for a risky change to a low-traffic path, because a canary at this volume cannot protect it
• Notes that a release outside working hours gives the gate even less traffic to judge

Follow-up questions

• How many requests does the canary see in its ten minutes?
• Every refund fails in the canary. Does the gate stop it?
• The normal error rate is 0.3%. What is the 1% threshold comparing against?
• How long would the canary have to run to see the refund failure, and is that the right fix?
• What should the gate look at besides the error rate?

Purpose

The document builds once and deploys by digest, which is right, and then updates both environments in one commit, which removes promotion altogether. The read is whether the candidate sees that staging now verifies nothing, and designs a promotion that is automatic without being blind.

Signals to score

• Notices that one commit updates staging and production, so both clusters take the change within minutes of each other and a failure in staging stops nothing
• Separates building once, which the document gets right, from promotion, which it leaves out
• Makes promotion a separate commit to the production directory carrying the digest that passed staging, made automatically after named checks or by a person
• Names what staging must show before promotion — a completed rollout, smoke tests against it, a period without a rise in errors — and what fails it
• Decides what happens when merges arrive faster than promotions: promote the newest digest that passed, never one that did not
• Restricts who and what can write to the production directory
• Says when production may change and what the path is for an urgent fix, given that controllers undo hand changes
• Uses the deploy repository's history to say what was running in production at a given minute
• Says deploying every merge to production still fits, as long as each one is verified first

Follow-up questions

• A merge breaks staging. When does production find out?
• The document says staging and production never drift. Should they?
• Three merges land within ten minutes. What gets promoted?
• Who or what may write to `envs/production`?
• What was running in production at 14:05 yesterday, and how do you know?

Purpose

Reads whether the candidate uses infrastructure code to make environments the same by construction, and whether they see the data, credentials, cost and clean-up that a per-pull-request environment brings with it.

Signals to score

• Reads the appendix and separates the difference someone chose — the replica count — from the two nobody did: the missing readiness probe and the lower memory limit in production
• Says what the missing readiness probe does in production: new pods get traffic before they can serve, on every rollout and every canary
• Defines each environment once with per-environment inputs, so an intended difference is a value and an unintended one has nowhere to live
• Refuses to put production's personal data in environments that every engineer and every pull request's code can reach, and proposes synthetic or masked data
• Notes that code from a pull request would run with credentials for that database
• Shares expensive pieces between previews — one database server with a database per pull request — rather than an instance each
• Plans for teardown failing: resources labelled with the pull request and an expiry, a scheduled job that destroys anything without an open pull request, and a cap on how many exist
• Asks which bugs the author expects realistic data to catch, and whether something cheaper would catch them

Follow-up questions

• Which of the three differences in the appendix did anyone choose?
• Production has no readiness probe. When does that hurt?
• Whose data is in last night's snapshot, and who can reach a preview?
• The teardown job fails one Friday. What does the account look like a month later?
• What bug is realistic data supposed to catch here?

Purpose

Not scored. The close gives the candidate what they need to judge the job; keep any observations in the notes.

Purpose

A specific, true gap is the most credible thing you can offer the engineer this round is written for, and fair warning to one who wanted a finished platform. Check it is still true before you say it.