Machine Learning Engineer interview questionsExperiment review — offline up, online down round

Purpose

Sets the contradiction as the subject and removes the option of spending time on definitions.

Purpose

Makes finding the buried fallback figure a scored act rather than luck. A candidate who read the whole dashboard says it here; one who read the headline does not, and will be led to it later at a cost.

Purpose

The load-bearing question. Anyone can list reasons an offline metric and an online result disagree; the separator is whether the hypotheses are ranked by the evidence already in the pack and whether each comes with the observation that would confirm or kill it.

Signals to score

• Names the fallback rate tripling as a hypothesis in its own right — a slower model served less often — and ranks it first or explains why not
• Ties the 40 ms latency increase to the fallback increase and to the mobile segment before treating them as three separate facts
• Names the offline evaluation's dependence on v6's exposure policy: v7 was scored on impressions v6 chose to show, so it was never scored on the items it would rank differently
• Names the gap between the offline metric and the business metric — AUC scores every pair, click-through happens at the top of the list — and checks whether NDCG@10 moved at the positions users actually see
• Says what to log for each hypothesis: model version and fallback flag per impression, per-position click-through, latency per request, and whether the item shown was in the training distribution
• Checks sample ratio and assignment before believing the online result, and says how
• Considers a feedback-loop or novelty effect — v7 shifting toward popular or toward unfamiliar items — and names the segment cut that would show it
• Distinguishes hypotheses the pack can already settle from ones that need a new log line, and says which is which
• Refuses to pick a winner before the fallback figure is explained
• Asks whether the two-week offline window and the nine-day online window overlap a promotion, a release or a holiday

Follow-up questions

• Requests served by the fallback ranker went from 0.4% to 3.1%. Is that a hypothesis or a fact, and what does it do to your list?
• v7 was evaluated on impressions that v6 chose to show. What did it never get scored on?
• The drop is returning users on mobile. Which of your three does that support?
• You want to log per-position click-through. Why is that not already in the pack?
• What would you have to see for "v7 is a worse model" to be the right conclusion?

Purpose

Tests metric literacy at the level the job needs: not the definition, but what the metric rewards, what it is blind to, and which alternative is fit for the decision being made.

Signals to score

• States what AUC rewards — correct ordering across all pairs of positive and negative — and what a ranker is judged on — a handful of positions per request
• Names at least one metric computed per request over the top positions, and says why per-request matters for a ranker
• Distinguishes ranking quality from calibration and says which one the downstream product depends on
• Says that any offline metric computed on logged impressions inherits the logging policy's exposure, and names a counterfactual or off-policy approach, or interleaving, as the way past it
• Proposes an offline metric that is aligned with the online metric the business reads, and says how they would check the alignment on past experiments
• Recommends holding out by time rather than at random, and says what a random split leaks for a ranker trained on ninety days of impressions
• Says what a three-point AUC gain would have to look like at the top of the list to be worth an online test at all
• Names the metric they would gate a launch on and the one they would merely report

Follow-up questions

• AUC up, click-through down. Can both be true with no bug anywhere?
• What is the smallest offline change you would bother testing online, and in which metric?
• Your training data is the last ninety days and your evaluation set is a random 10% of it. What is wrong with that for a ranker?
• Is calibration part of this model's job? Who consumes the score?
• You have thirty past experiments with offline and online results. What do you do with them?

Purpose

Tests whether the candidate can act under an ambiguous result and whether they have internalised that the rule comes before the data. Most candidates can argue any of the three actions; few can say what was missing on day zero.

Signals to score

• Does not ship v7 as tested and says why in one sentence
• Does not simply kill it either, and separates "the model" from "the deployment" in the decision
• Proposes the concrete next step: fix or raise the timeout, or serve v7 only where it meets latency, and re-run with the fallback rate as a guardrail
• Names a guardrail set — fallback rate, p95 latency, sample ratio — that should have stopped the test automatically, with thresholds
• States the primary metric and the minimum detectable effect the test was powered for, or says that nobody wrote them down and that this is the actual finding
• Says how long the test needed to run for the click-through interval to be trusted, and whether nine days covers a full weekly cycle
• Names who owns the decision and what they agreed to before day one
• Says what they will write down for the next test, and it is a rule with numbers rather than a lesson
• Considers the cost of extending — users in the treatment arm are seeing a worse product — and weighs it rather than ignoring it

Follow-up questions

• You extend by a week and click-through is still down 1.8%. Now what?
• Who owns this decision, and did they agree to anything before the test started?
• What should have stopped this test on day two?
• Nine days. Does that cover what it needs to cover?
• What is the one sentence you write in the experiment template so the next person does not end up here?

Purpose

A candidate who has run experiments asks about pre-registration and guardrails; one who has not asks about the experimentation platform. Naming the topics makes the signal about the choice.

Purpose

Ends on a specific, true, unflattering fact — the strongest pitch to the candidate this round exists to hire and a deterrent to one who wanted the process already fixed. Check it is still true before you say it.