AI Engineer interview questionsEvals & Measurement round

Purpose

Separates candidates who diagnose from candidates who prescribe. The entire discriminator is whether a proposed fix arrives before a partition of the failures does.

Signals to score

• Reads the retrieved chunks before commenting on any of the answers
• Partitions the six into named failure categories out loud instead of walking them one at a time
• Distinguishes at least three of: document absent from the index, indexed but outside top-k, inside top-k but buried under higher-scoring noise, retrieved and ranked well but ignored by the generator
• Cites specific trace indices as evidence when naming a category
• Asks what the indexing job actually ingested before assuming the source document exists
• Counts the categories by frequency and states which one they would work on first
• Withholds any fix until after the partition, or explicitly labels a fix as a hypothesis with a stated way to test it
• Names what the regenerate filter selects for and what it systematically misses — it catches answers the user could see were wrong and never catches the confidently wrong ones the user believed, which is the more expensive class
• Names at least one trace where retrieval was correct and generation still failed, and questions at least one labelled "correct" answer

Follow-up questions

• In the trace where the answer chunk ranked second and the generator ignored it — note which index that is before the round — retrieval failure or generation failure?
• If I told you the source document was never in the index at all, which of your categories is that, and does raising k help?
• You have grouped them. Which group is largest, and what would you have to see to be wrong about that?
• Suppose I handed you a thousand of these instead of six. What changes about how you do this, and what stays the same?
• What are you writing down while you read, and where do those notes end up?

Purpose

The most reliable separator in the round. People who have run a judge in production reach for its error rates unprompted; people who have read about judges list biases and stop there. Read the arithmetic in the model answer before you run this — you will be asked to defend it.

Signals to score

• Asks whether the nightly job actually ran, since an unchanged number for three weeks is as often a cached artifact or a silently failing job re-posting yesterday's file as it is a stable system
• Asks whether the judge has ever been scored against human labels before discussing anything else
• Asks for true positive and true negative rate on a held-out labelled set instead of a single "judge accuracy"
• Corrects the observed 94% using the judge's own error rates rather than taking it at face value
• Names at least two of position bias, verbosity bias or self-preference without being prompted, and separately raises that the judge's own behaviour moves when its prompt or model changes
• Flags judge and generator sharing a model family as a self-preference risk
• Asks whether the 300 samples come from live traffic or a frozen set, and whether traffic has shifted
• Takes a position on whether the judge may gate and attaches conditions to it — either it never gates and instead ranks a human review queue, or it gates only on failure modes with measured high agreement, on a paired comparison against the current production prompt, at a threshold derived from run-to-run spread and not from an absolute score. The unconditional answer in either direction is the weak one
• Gives a number for how many human labels they would want, with a rationale for that number, and names a person who would produce them
• Proposes a recurring human audit of a small sample instead of a single one-off validation

Follow-up questions

• Suppose nobody has ever compared this judge to a human. What is the first thing you do this week?
• The judge scores output from its own model family. What does that do to the number?
• We want to fail the build when the judge score drops. Talk me into it or out of it.
• How many human labels would you need before you believed the judge, and who produces them?
• Someone reordered the fields in the judge prompt last month. Does the three-week trend still mean anything?

Purpose

A deliberately bad instruction. The signal is what the candidate does with an instrument they have been handed, and whether their objection is grounded in annotator agreement or in taste.

Signals to score

• Does not simply hand over five bands: either declines the scale, or produces it alongside the instrument they would actually ship and says which one they would defend
• Grounds the objection in instability between adjacent points across annotators and across weeks
• Proposes binary pass/fail against named, enumerated failure modes
• Gets granularity by decomposing into several binary checks instead of lengthening the ruler
• Offers pairwise comparison against a frozen reference output when the requester genuinely wants one trending number, names win rate as the thing to trend, and knows the trade it is making — easier for annotators to agree on, sensitive to presentation order, so slots get randomised or swapped
• Reaches for a chance-corrected agreement statistic when two people label, and names one
• Asks who will label, how often, and on what sample before designing anything
• States what decision the resulting trend line would actually drive
• Still gives the requester a dashboard number, such as pass rate per failure mode
• Warns that a suite sitting at or near 100% has stopped carrying information

Follow-up questions

• Two annotators score the same response 3 and 4. What do you do with that pair?
• I still want one number on a dashboard. What number do you give me instead?
• Percent agreement between your two labellers is 85%. Is that good?
• Your suite has been at 100% pass for a month. Good news?
• Where do the named failure modes come from in the first place?

Purpose

The first half has a widely believed half-answer, which makes the follow-up impossible to bluff. The second half is the decision the folklore actually costs money on.

Signals to score

• Checks the mundane causes before the exotic one — same model version behind the alias, same request parameters, prompt-cache state — and only then reaches for a mechanism
• Names batch-size dependence as that mechanism: identical requests hitting kernels that reduce differently at different batch sizes. "Floating-point addition is not associative" on its own is the incomplete answer, and capacity-limited MoE expert routing is an equally legitimate batch-dependent mechanism
• Notes that output therefore depends on concurrent server load, since batch composition varies with traffic
• Adds that this is now fixable at a performance cost, since batch-invariant kernels have shipped in at least one major serving stack — a credit, not a contradiction
• Asks for n before evaluating the three-point delta
• Recognises the comparison is paired — same items, two prompts — and asks for the discordant pairs instead of two aggregate scores
• Asks for the spread across repeated runs of the same prompt, not only across the two prompts
• Asks whether the eval set itself changed between the two runs
• Raises unprompted that v2 may have been written while staring at the examples it is now scored on
• Proposes a decision rule set in advance: what delta, at what n, would justify shipping

Follow-up questions

• If sampling is not the source, what in the serving stack could change the result?
• Would the same request give the same answer at 3am and at peak traffic? Why might it not?
• Run v1 against itself twice. What spread do you expect, and how would you find out?
• The three points came from an n of 40. Does that change your answer?
• v2 was written by reading the ten examples v1 failed. What is the problem with the score now?

Purpose

Tests whether the eval set is treated as a living artifact with an owner and a decay rate, and whether the candidate checks the denominator before theorising about the model.

Signals to score

• Asks whether escalations are up 40% in absolute count or as a share of sessions, and whether the escalation path or the SLA changed, before drawing any conclusion about the model
• Checks whether the model behind the API alias moved during the window
• Names distribution shift between the eval set and live traffic as the leading hypothesis once the signal survives those checks
• Proposes sampling real production traces instead of reasoning about what might have changed
• Re-runs error analysis on those traces with a stated stopping rule, and expects new failure categories rather than more of the old ones
• Folds the new failure modes back into the set, names the owner and the cadence, and versions the set alongside the prompt it scores
• Gives a size for a set they actually maintained and justifies it against two costs: what it took to label, and the smallest quality delta that size can resolve. A number with no justification either way — 50 or 5,000 — is the weak answer
• Raises contamination: an eval example that has leaked into a few-shot block or a fine-tune
• Names implicit production signals they would instrument — regenerations, manual edits, abandonment, escalation to a human

Follow-up questions

• What is in the eval set that is not in production traffic, and what is in production that is not in the set?
• How big is the set you would actually maintain, and when do you stop adding to it?
• Someone moved three eval examples into the prompt as few-shot demonstrations. What happened to your numbers?
• The last time you read a batch of outputs by hand, what did the notes look like and where did the categories end up?
• Nobody clicks thumbs-down. What do you measure instead?

Purpose

Forces both halves of the trade into one answer. Cost work with no quality metric and eval work with no bill are each half an answer, and most candidates volunteer exactly one half. Three asks in nine minutes is already tight — do not add the fourth.

Signals to score

• Asks whether the p95 target is time-to-first-token or the completed response before naming a single lever, since the two have almost disjoint fix lists
• Tiers the suite by cadence: deterministic checks per commit, sampled judge nightly, full corpus weekly or at release
• Produces an actual dollar figure, or a way to compute one from tokens, price and sample size
• Gives a wall-clock number and treats CI duration as a real constraint on suite design
• Picks a smaller model as judge and justifies it by measured agreement, not by price alone
• Names model tiering by task — a classification or routing step that does not need a frontier model
• Names caching with a target hit rate instead of as a general good idea, and can say how a bad cache would show up
• Proposes cutting context before growing it, and can say what specifically gets cut
• Separates format-failure retries from wrong-answer retries when reaching for structured outputs
• Names the quality metric they will watch while making the cut, and the movement that would make them stop

Follow-up questions

• You want to judge every pull request with the largest available model. What does that cost a month?
• Which step in your pipeline does not need a frontier model, and how would you prove it?
• You cache aggressively and quality drops. How would you have found out?
• Where does streaming genuinely help, and where is it cosmetic?
• Latency target met, cost target met. How do you show me quality did not move?