Interview rubric: what Yes, Not sure and No sound like for six skills
An interview rubric says what the interviewer listens for and how to read what they hear. In this one, each of six skills has three signals quoted from an interview plan, and each signal is read Yes, Not sure or No, with an example of each. The round ends in one of four recommendations. Nothing is scored, weighted or averaged.
Updated . Written by Hirezen. The signals are quoted from the plans. What Yes, Not sure and No sound like is written for this page as an example; your candidates will say it their own way.
Six skills, three signals each, every reading with an example, and the four points.
How this rubric is read
A skill, or competency, is what the hire is decided on, such as Ownership. A signal is one thing the interviewer listens for under one question, written so that it is either heard or not. The interviewer marks each signal as the answer comes, and the recommendation is a judgment on the round, not a sum of marks.
- Skill
- What the hire is decided on, named on the position; many teams call it a competency. In Hirezen an interview checks two skills at most, and the loop flags a must-have that fewer than two rounds read.
- Signal
- One thing to listen for under one question, such as “Names the role at the customer who owned it after the handback.” Written so that two interviewers would mark the same answer the same way.
- Reading
- Yes, Not sure or No, marked during the interview, with the minute it was marked. Not sure is a reason to ask a follow-up, or to write the question down for the next round.
- Recommendation
- One of four, from Strong No to Strong Yes, chosen when the interview ends. A judgment on what the round is for, informed by the readings rather than counted from them.
- No score
- There is no score, weight or average. Three Yes and a No are four things heard, not 75%, and the debrief reads every interviewer’s write-up side by side.
- Yes: the answer contained it.
- Not sure: you could not tell — a reason to ask, or to write it under Would settle it.
- No: the answer did not, or showed the opposite.
Coding
Read in the AI Engineer coding round, where a function that calls a model works on the happy path and the candidate makes it safe to put in front of a customer.
The question
“The function works against the happy path. Make it safe to put in front of a customer. Start wherever you think the biggest hole is, and tell me why that one.”
| Signal | Yes | Not sure | No |
|---|---|---|---|
| Validates the model's output against a schema at the boundary rather than trusting its shape | “Nothing downstream touches this until it validates against the schema.” | Adds a type for the response, but never says what happens when the model ignores it. | Reads the summary field straight off the model’s JSON. |
| Treats a parse failure as an expected outcome with a policy, not as an exception to log | “If it doesn’t parse, we serve the last good summary and send the ticket to the review queue.” | Catches the error and says “we’d handle it”, with no policy named. | Logs the exception and returns null. |
| Keeps the failure path returning the same type as the success path, rather than null or a thrown error the caller must know about | Returns the shape the caller already handles, with a flag that says it is a fallback. | Returns an empty string, which type-checks but which the caller cannot tell from a real summary. | Throws, and leaves every caller to find out. |
Production judgment
Read in the last question of the AI Engineer caching round: a customer has been shown another account’s quota by the cache, and it is 10am.
The question
“A customer emails support to say the assistant told them how much quota was left on an account that is not theirs. It is 10am. What do you do in the next hour, and what do you change permanently?”
| Signal | Yes | Not sure | No |
|---|---|---|---|
| Turns it off before diagnosing, and can say whether that is a flag or a deploy | “First the cache goes off. It’s a flag, so that’s a minute. Then we work out why.” | Says it would be turned off, but not how, or how long that takes. | Starts reading logs for the bad entry while the cache keeps serving. |
| Treats it as a data incident rather than a quality bug, and says who else needs telling | “A customer saw another account’s data. Security and whoever owns customer notices hear about it this morning.” | Calls it serious, but keeps it inside the engineering team. | Files it as a wrong answer to fix next sprint. |
| Fixes it permanently at the classification gate, not by raising the threshold | “Account-specific questions never reach the shared cache, whatever their score.” | Blocks quota questions only, without a rule for the whole class. | Raises the similarity threshold and calls it fixed. |
Incident response
Read in the SRE incident review: a rate limit pushed to every region took 99 minutes to recover from, and the candidate reads the timeline as the incident commander should have run it.
The question
“Read the timeline as the incident commander should have run it. Where did the 99 minutes go, and which of them could have been saved on the day — not by next quarter's engineering, but by decisions made in the incident channel?”
| Signal | Yes | Not sure | No |
|---|---|---|---|
| Names "what changed just before this started?" as the question that should have come first after declaring, and points at the 14:00 push lining up with the 14:02 graph | “The first question after declaring is what changed. The 14:00 push lines up with the 14:02 graph.” | Mentions the push, but only when the timeline reaches it. | Spends the first minutes on whether the customers were over their limits. |
| Says a recent change is rolled back first and proven guilty afterward, and that waiting 13 minutes for the author's confirmation was a choice | “You roll back the recent change and prove it guilty later. Waiting 13 minutes for the author was a choice.” | Agrees the rollback was slow, without saying it should not have waited. | Defends waiting for the author to confirm the change was at fault. |
| Asks when the 50 largest customers were told, and says they should have heard before the root cause was known | “When did the fifty largest customers hear? Before we knew the cause, it should have been.” | Mentions a status page, but not who heard or when. | Leaves customers to the postmortem. |
Troubleshooting from evidence
Read in the network administrator incident review: a copy from a file share to a branch stalls while a copy to it works, and the candidate has packet captures from both ends.
The question
“Tell me why a copy from the share stalls when a copy to it works — and show me the packets in these captures that prove it.”
| Signal | Yes | Not sure | No |
|---|---|---|---|
| Uses the direction of the failure as evidence: small exchanges succeed and only large transfers towards the branch fail | “Small exchanges work and only big transfers towards the branch stall, so it’s packet size, one way.” | Notices that uploads work, but does not use it to narrow anything. | Treats it as the share being slow. |
| Dismisses the bad checksums as an artifact of capturing on the sending host with offload | “The bad checksums are the capture: it was taken on the sender, and the card fills them in afterwards.” | Sets the checksums aside without saying why they can be. | Chases the checksums as the cause. |
| Proposes the DF ping test at 1372 and 1373 bytes, and predicts both results before seeing them | “Ping with don’t-fragment set at 1372 and at 1373. The first gets through and the second doesn’t.” | Suggests a ping test, with no size and no prediction. | Proposes replacing the router before testing anything. |
Ownership
Read in the last question of the forward deployed engineer’s engagement deep dive: what was still running three months after they left, who owned it, and how they know.
The question
“What was still running three months after you left, who owned it, and how do you know?”
| Signal | Yes | Not sure | No |
|---|---|---|---|
| Names what was still running, specifically, and what had been switched off or abandoned | “The nightly reconciliation job still runs. The dashboard we built for the CFO was switched off in month two.” | Says most of it is still in use, without naming what. | Talks only about what was delivered, not what survived. |
| Names the role at the customer who owned it after the handback | “Their finance systems lead owns it. We handed the runbook to that team in week nine.” | Names a team, but no role or person. | “Their IT, I think.” |
| Knows because they checked - a message, a metric, a visit - and says when | “I asked their lead in March and read the job’s run history. It hadn’t missed a night.” | Heard from someone there that it is still used. | Assumes it is running because nobody complained. |
Prioritization
Read in the SEO manager’s budget defense: an executive is cutting search by 60% and has asked what the candidate would cut.
The question
“[In character] Let's say you've made your point about forecasts. It doesn't change my arithmetic. Make the case. And be aware I'm going to ask you what you'd cut, so if the answer is nothing, save us both the time.”
| Signal | Yes | Not sure | No |
|---|---|---|---|
| Concedes a specific, substantial piece of the budget before defending anything | “Cut the informational articles an assistant now answers in full. That’s about a third of the spend.” | Offers a cut too small to change the executive’s arithmetic. | Defends the whole budget. |
| Offers a phased cut with a checkpoint, rather than a binary | “Half the cut now, the rest in two quarters if the transactional pages hold.” | Asks for more time, with no date and no test. | Treats it as all or nothing. |
| Quantifies something, even roughly, and flags the uncertainty rather than hiding it | “Roughly 40% of organic revenue lands on those pages. Take that as plus or minus twenty points.” | Gives a number with no range and no source. | Argues without a number. |
The four-point recommendation
Every interviewer ends the round with one recommendation: Strong No, No, Yes or Strong Yes, each with a fixed definition so that two interviewers who both pick Yes mean the same thing. There is no middle. A doubt is written down as the question the next round could ask, not cast as a vote for unsure.
- Strong No
- Clear gaps on what this round is for
- No
- Below the bar for this round
- Yes
- Meets the bar; I would work with them
- Strong Yes
- I would argue for them in the debrief
No middle: if you are torn, pick the side you lean to and say below what would settle it. The grid of skills against rounds for a whole loop, with the four points in its last row, is the scorecard on the after-the-interview page.
Questions
- What is an interview rubric?
- A written guide to how an interviewer reads what they hear: the skills, or competencies, the round assesses, what to listen for under each question, and what a strong, unclear and weak answer sounds like. It makes two interviewers’ judgments comparable. Here, each skill has signals read Yes, Not sure or No, and the round ends in one of four recommendations.
- What is the difference between a rubric and a scorecard?
- A rubric says how to read an answer. A scorecard is where the reading is recorded: one per interviewer per round, with a mark for each skill or signal and an overall recommendation. The rubric is used during the interview; the scorecard is filled in and read at the debrief. Hirezen’s write-up is the scorecard, and the plan’s signals are the rubric.
- Should an interview rubric use a 1 to 5 scale?
- It can, but a number invites an average, and an average hides the disagreement the debrief exists to find. Three readings per signal are easier to agree on than five points, and a recommendation with four points and no middle forces the interviewer to say which way they lean. If your ATS needs numbers, map the four points to it once.
- How many skills should one interview assess?
- One or two. An hour holds two skills asked about properly, with follow-ups, and an interviewer rating three or more is rating more than they heard. Spread the rest across the loop so that every must-have is read in at least two rounds by different people. Hirezen lets an interview check two at most.
- What does Not sure mean on a rubric?
- That the interviewer could not tell from the answer, not that the answer was average. It is a prompt to ask a follow-up while the candidate is still in the room, or to write the open question down so the next round can settle it. A rubric with no Not sure pushes interviewers to guess.
- Can I use this rubric for roles outside engineering?
- The method, yes: skills, signals written so they are either heard or not, three readings and four recommendation points work for any role. The signals on this page come from technical, operations and marketing rounds, so for another role write signals from that role’s own questions, one observable thing each.
What a write-up holds once the interview ends, and how the debrief reads every interviewer’s side by side, are on the after-the-interview page; nine filled-in write-ups are on interview feedback examples; how a plan’s questions and signals are written is on how plans are structured.
Use the rubric in the interview, not after it
Every plan in the library carries its signals under each question. Mark them Yes, Not sure or No as you hear the answer, and your readings are already in the write-up when the interview ends. No bot joins the call and nothing is recorded.