DevOps Engineer interview questionsCoding Test round

Purpose

Makes reading the first task and names the incidents, so a candidate who prepared for an algorithm spends the opening minutes on the script, and the diagnosis has something specific to explain.

Purpose

Allows lookups and either language, and warns that the environment will change underneath them, so time goes on the script rather than on doubting their own code.

Purpose

Anyone can list shell hygiene. The read is whether each incident is traced to the line that caused it, through what the shell and curl actually do with an exit status, and whether the rest of the list is ordered by harm rather than by line number.

Signals to score

• Traces incident one to the pipe into `tee`: without `pipefail` the pipeline returns `tee`'s status, so a failed build carries on and pushes the `:latest` the runner already had
• Says why the long-lived runner matters — a fresh one would have had no local `:latest`, and the push would have failed
• Traces incident two to `docker tag` running after `docker pull`, so `:previous` names the new build and `rollback.sh` restarts it
• Reads `curl -s` as succeeding on any HTTP response, so a host answering 503 passes, and notices the `until` loop has no limit
• Finds the token in the trace — the assignment and the login line both print it — and says it has to be rotated, not only hidden
• Notices that stop, remove and run leave a host serving nothing, and no container at all if `docker run` fails
• Notices that a re-run after that failure stops at the same host for a different reason: there is no container to stop
• Asks whether two deploys can run at once, since `:latest` can move between one host's pull and the next
• Orders the list by what can hurt production soonest, not by position in the file
• Says what is right: the env file, and one host at a time

Follow-up questions

• The build failed. Whose exit status did `set -e` look at?
• Would incident one have happened on a fresh runner?
• `rollback.sh` finished without an error. What did it start?
• What does `curl -s` exit with when the service answers 503?
• Is there anything in this repository you would treat as already leaked?

Purpose

Sets proof as the deliverable and admits the time limit, so the candidate has to choose an order rather than rush every fix half-way.

Purpose

The load-bearing exercise. Most of the fixes are a line or two each; the read is whether the candidate makes every claim of success checkable — an exit status that means something, a reference that cannot move, a health check that can fail — and tests each one by switching its failure on.

Signals to score

• Makes a failed build stop the script — `pipefail`, or no pipe at all — and shows it by running with `FAIL_BUILD=1` before touching anything else
• Tags the image with the commit SHA and deploys by the digest the push reports, so every host pulls the same image whatever is pushed meanwhile
• Requires a 200 from the health check — `curl -f` or an explicit status test — with a timeout on each request and a deadline for the host
• Takes the deadline from how long the service needs to start, and makes expiry a failure that names the host
• Drains each host from the load balancer before stopping its container, and enables it again only after the check passes
• Exits non-zero with the host, the digest it was deploying, and what that host is now running
• Passes the token with `--password-stdin` and keeps it out of the trace
• Tests each fix by switching on the matching failure in the fakes instead of reasoning about the code
• Says what they did not reach and the order they would continue in

Follow-up questions

• Run it with `FAIL_BUILD=1`. What does the log say now?
• Two deploys push within a minute of each other. Which image does `app3` get?
• What exactly makes your health check fail?
• How long do you wait for a host, and where does that number come from?
• `app2` is drained and its check never passes. What state is the load balancer in when the script exits?

Purpose

Reads whether the candidate's automation leaves state an operator can act on at 3am — a durable record of what was running, a rollback that deploys a known digest, and scripts that do the right thing when re-run half-way through.

Signals to score

• Records each host's running digest before changing it, somewhere that survives both the host and the runner, not as a tag on the host
• Rolls back by recorded digest, through the same drain, check and enable as the deploy
• Decides what `app2` failing means for `app1` — roll it back too, or stop with a known mixed fleet — and says why for this service
• Makes a re-run skip hosts already serving the target digest and treat a missing container as a normal state
• Handles a new host with nothing running, where there is nothing to record
• Keeps two deploys from running at once, with the CI's concurrency control or a lock, and says what the second one should do
• Gives the rollback a dry run that prints what each host runs now and what it would change
• Writes the test: `HEALTH_APP2=503`, deploy, rollback, then asserts from the fakes' state that every host is back on its recorded digest

Follow-up questions

• Where does the record live, and what happens to it when the runner is rebuilt?
• `app2` failed. What is `app1` running right now, and is that what you want?
• Run the deploy twice with nothing changed. What happens the second time?
• A second merge starts a deploy while this one is on `app2`. What happens?
• It is 3am and someone else has to run your rollback. What do they see before it changes anything?

Purpose

Separates what a platform does from what a release still needs. Someone who has run rollouts on an orchestrator knows its rollback restores a pod template and nothing more; someone who has only read about it believes rollback is solved.

Signals to score

• Names what the orchestrator takes over: replacing instances a few at a time, waiting for readiness, keeping earlier revisions to roll back to
• Says a readiness check is only as honest as the endpoint behind it, so a shallow `/healthz` moves the 503 problem into the probe
• Keeps liveness checks off dependencies, so a database outage does not restart every pod
• Keeps build-once and deploy-by-digest in the pipeline, because the orchestrator runs whatever reference it is handed
• Knows that rolling back a deployment restores its pod template only, not a config map or secret changed in place
• Knows whether their orchestrator rolls back a stalled rollout by itself — a Kubernetes Deployment reports failure and stops — and says what should decide
• Keeps a deploy record outside the cluster, because revision history is short and records no who or why
• Says two pipeline runs racing to deploy are still the pipeline's problem

Follow-up questions

• Which parts of today's script does the orchestrator make unnecessary?
• Your readiness probe calls `/healthz`. What does `/healthz` check?
• A rollout gets stuck half-way. What happens next, and who decides?
• You roll back the deployment. What is still new?
• Where do you look to find out what was running last Tuesday at 10:00?