DevOps Engineer interview questionsTechnical Interview round

Purpose

Tells a candidate expecting a coding test or a quiz that the hour goes on one running system, and that the last part is live.

Purpose

Lets the candidate read in silence and warns, without saying which, that some lines are healthy — so leaving the memory settings and the deadline alone is the candidate's own judgement.

Purpose

Container runtime, read through a resource limit. The limit, the thread count and the throttling figure sit on separate lines; the read is whether the candidate joins them into one mechanism instead of taking 0.42 cores as headroom.

Signals to score

• Takes the throttled periods, not the 0.42-core average, as what needs explaining
• Explains the quota: a 1-CPU limit gives the container's threads 100 ms of CPU between them per 100 ms period, then all of them wait for the next
• Connects `worker threads: 16` to the node's 16 CPUs, and works out that sixteen busy threads spend the quota in about 6 ms
• Explains why p50 held and p99 did not: a request finishing while the quota lasts is untouched, one caught when it runs out waits for the next period, sometimes more than once
• Sets the five-second carrier calls aside: at 0.3% they sit beyond the 99th percentile
• Says doubling the replicas lowers the average without changing the bursts, for twice the requested CPU
• Matches parallelism to the CPU allowed — worker threads sized from the limit if they don't block on I/O, or a limit that fits the threads
• Weighs dropping the CPU limit and keeping an honest request: what the request still guarantees on a busy node, and what an unlimited pod can take from its neighbours
• Keeps the memory limit: memory cannot be throttled, and a container over its limit is OOM-killed
• Judges the fix by throttled periods and p99 at the next evening peak, not by average CPU

Follow-up questions

• The average is 0.42 cores against a limit of 1. Where did the rest of the limit go?
• Why did the p50 barely move?
• The pull request doubles the replicas. What does it do to the p99, and what does it cost?
• Take the CPU limit off. What stops this pod taking a neighbour's CPU?
• Would you take the memory limit off as well?

Purpose

Tests whether the candidate reads "every pod takes 30 seconds" and "no pod logged `draining`" as one fact, and knows the signal path well enough to see what the obvious fix leaves broken.

Signals to score

• Reads the 30 seconds and the missing `draining` line together: the handler never ran, and every pod was killed at the default grace period
• Finds the cause in the entrypoint: the shell is PID 1 and alone receives SIGTERM, which a PID 1 without a handler ignores
• Ties the bursts to the kill: `checkout`'s long-lived connections stay on the old pod, still serving, until SIGKILL cuts them mid-call
• Explains why the storefront is spared: the ingress picks a pod per request and stops picking a terminating one within a second or two
• Fixes the signal path with `exec` on the last line — under a minimal init if the service can't reap children — and knows an init in front of the unchanged script is not the fix: by default tini signals only the shell, and an init that signals the whole group still exits with the shell and kills the service mid-drain
• Predicts the next failure: the service will stop accepting the moment it hears SIGTERM, while its removal from the endpoints is still reaching the nodes and the ingress
• Orders the shutdown: a few seconds still serving, then stop accepting, tell gRPC clients to reconnect elsewhere, finish in-flight calls, exit
• Sizes the grace period from that sequence with room to spare, so the kill never fires in a normal shutdown
• Proves it with a rollout under load: failed calls per caller, terminations in seconds, `draining` in every old pod's log

Follow-up questions

• Every old pod takes 30 seconds to go. What is it doing for those 30 seconds?
• Why does `checkout` see errors when the storefront doesn't?
• A pull request adds `exec` to the entrypoint's last line. Once it merges, what fails next?
• How long should the grace period be, and what is it made of?
• How will you know the next rollout was clean?

Purpose

A distribution with a gap between 1.5 and 5 seconds is a timer firing, not a slow server. The read is whether the candidate derives the lookup volume from `resolv.conf` and the call rate, and fixes what causes the lookups before scaling what answers them.

Signals to score

• Reads calls just over five seconds, with almost none between 1.5 and 5, as a lost packet retried on a fixed timer, not a slow carrier
• Names the resolver's five-second retry as the likely timer, and knows a lost TCP connection attempt would show at about one second
• Connects `checkout`'s 0.3% of DEADLINE_EXCEEDED to the same calls, and leaves its deadline alone
• Counts one lookup from `resolv.conf` — three search domains first, A and AAAA each, eight queries — and matches 1,100 lookups a second to the 9,000 the cluster DNS answers
• Traces the lookups to a new connection per call, and says reuse removes most of them and a TCP and TLS handshake from every call
• Lowers the cost of a lookup — a smaller `ndots` in the pod's DNS config, or a fully qualified name with its trailing dot tested against the certificate check and Host header
• Asks where queries are lost before adding DNS replicas: the DNS pods' throttling and errors, the nodes' connection-tracking counters
• Proposes a DNS cache on every node, answering repeat queries locally and outside connection tracking
• Times DNS separately in the client, so the next slow lookup is labelled as one
• Caps how long a reused connection lives, so the service notices when the carrier's address changes

Follow-up questions

• Why five seconds, and not one or three?
• How many DNS queries does one carrier call cost today?
• Which single change removes the most of those queries?
• Would you add DNS replicas?
• Once connections are reused, when does the service notice that the carrier has moved?

Purpose

Opens with what an engineer called in would know — a symptom, a number, one recent change — so that everything else has to be asked for.

Purpose

Reads how the candidate narrows a live fault — who is failing, where, since when — before touching anything, and whether their first move changes where `checkout`'s calls go or only adds pods those calls will never reach.

Signals to score

• Establishes who is failing before changing anything: `checkout`'s gRPC calls, while the storefront is slow only on some pods
• Asks for per-pod numbers, not the average, and finds five pods at their limit beside thirteen nearly idle
• Reads the autoscaler's 81% as pods at 200% and 36% of request averaged together, and says more replicas cannot help
• Explains the idle new pods: each `checkout` pod holds one long-lived HTTP/2 connection, routed once when it opened, and every call since has used it
• Asks what changed, and takes "nothing but traffic" as evidence that the imbalance predates the surge
• Works out that one connection's 250 calls a second needs more than a pod's one CPU, so moving connections cannot fix it
• Picks a first move that cuts the load per connection or adds CPU behind it — more `checkout` pods, or more CPU for the busy five — and says what it costs
• Rules out the moves that only look like action: restarting `checkout`, deleting the busy pods, raising the autoscaler's maximum, a longer deadline
• Confirms the first move on deadline failures and per-pod CPU before making a second
• Fixes it next week with per-call balancing — round-robin over a headless Service, or a proxy that balances requests — and a maximum connection age on the server

Follow-up questions

• Twelve pods became eighteen. Why are the six new ones idle?
• The autoscaler says 81% against a target of 80%. Is it wrong?
• You restart all six `checkout` pods. Where do their calls go?
• What is your first move, and how will you know within five minutes whether it worked?
• What would have shown you this before the promotion?