Summary slide. Follow one consequential comment from the original change through its support, response, and resolution. Save the slide (PNG)
Thirty comments about naming can reveal less judgment than two well-supported comments about an authorization gap. Assess code review samples by tracing consequential comments from the original change through reasoning and resolution, not by counting them.
Consider a hypothetical hiring decision. One candidate submits extensive formatting feedback; another identifies a cross-organization access problem and explains how to test the fix. The second sample offers stronger evidence of risk recognition, but only if the supplied context supports the finding.
A comment is a claim, not proof of judgment.
The assessment has two parts: how well the engineer reasoned, and whether you have enough evidence to judge that reasoning. Keep them separate.
Ask for a review packet, not praise
A useful packet lets you reconstruct a small engineering decision. It need not include an entire repository.
Ask for these elements, subject to authorization:
| Evidence | What it helps establish |
|---|---|
| Authorized context | The change’s purpose, constraints, and what the candidate may share |
| Original change | The version reviewed, with enough surrounding behavior to evaluate the comments |
| Candidate’s feedback | Their contribution, distinguished from other reviewers’ comments |
| Author’s response | Whether the feedback was clarified, challenged, accepted, or deferred |
| Revision or disposition | What changed, what remained unresolved, and any verification |
Include a short note about the candidate’s role and review scope. A review limited to an interface change cannot establish that they assessed the whole system’s security.
Public visibility does not authorize disclosure of private context. Candidates should remove secrets, customer information, and restricted details, and share only authorized material. If safe redaction makes the example unintelligible, offer a structured walkthrough instead.
Distinguish findings, questions, and unsupported objections
Classify consequential comments before scoring them. Confidence alone does not establish validity.
A finding connects a condition to a supported consequence
For example: “This lookup uses the supplied invoice ID without restricting it to the requester’s organization. In the synthetic fixture, an organization A user can retrieve organization B’s invoice.”
That is a finding only if the packet establishes the relevant request path and authorization behavior. An isolated query may be insufficient: another layer could enforce access.
A question identifies an unresolved assumption
For example: “Does an earlier layer restrict invoice access by organization? I cannot see that check in this diff.”
This can demonstrate judgment even when the answer confirms that another layer handles access. Assess whether the question was proportionate to the risk and available context, not whether it produced a code change.
An unsupported objection asserts risk without establishing it
For example: “This must use a different query library because the current one is unsafe.” Without a failure mechanism, applicable requirement, or supporting evidence, that objection does not establish a defect.
Preferences can still help. The problem is treating one as a blocking correctness issue without justification.
A resolved question can be better review work than an unjustified demand.
Use the Review Trace rubric
Rate four dimensions using these anchors. They are a shared starting point, not a universal hiring threshold.
| Dimension | 0: Weak in observed evidence | 1: Partly demonstrated | 2: Clearly demonstrated |
|---|---|---|---|
| Issue validity | Claims a defect contradicted by context or without support | Identifies a plausible concern but leaves a key assumption unresolved | Establishes a finding or frames a justified question with appropriate uncertainty |
| Prioritization | Treats a preference as blocking without justification or misstates the risk | Recognizes significance but leaves urgency or trade-offs unclear | Connects impact and likelihood to a proportionate recommendation: block, investigate, or defer |
| Actionable reasoning | Gives a verdict without enough explanation to respond | Explains the concern partly but leaves the next step ambiguous | Explains the mechanism and offers a concrete check, constraint, or remedy |
| Follow-through | Disregards relevant follow-up evidence or closes the issue without justification | Engages with the response but checks resolution incompletely | Updates their view when warranted and checks that resolution addresses the concern |
Use not observable, rather than 0, when the packet cannot support a judgment. Without an author response or revision, follow-through is not observable.
Give each dimension a separate evidence label:
- Sufficient: Relevant context supports the rating.
- Partial: Some support exists, but a named gap could change the assessment.
- Unavailable: The packet does not support a rating.
For example: “Actionable reasoning: 2, sufficient evidence. Follow-through: not observable, evidence unavailable.”
Missing evidence is not evidence of poor performance.
I would not total these ratings into a leaderboard. Compare demonstrated strengths with the role’s requirements, then use gaps to choose follow-up questions. One review supports a narrower conclusion than consistent performance across an engineering role.
Worked example: an invoice export review
This packet is hypothetical. All records and tests use synthetic data in an isolated test environment.
Assumed context: An export endpoint must return invoices only from the authenticated user’s organization. The supplied handler, called functions, and tests show a lookup by request-supplied ID. No organization access check occurs before the export is returned.
The candidate leaves three comments:
- Finding: “The lookup is not restricted to the requester’s organization. With these fixtures, user A can request invoice B. Please enforce the organization boundary and add a cross-organization denial test.”
- Question: “Should unauthorized users receive the same response as users requesting an unknown invoice? I do not see a requirement.”
- Preference: “Could we rename
resulttoinvoiceExport? Non-blocking.”
The author adds an organization-scoped lookup and a cross-organization test. The candidate notices that the test checks only response format, so it would also pass against the original implementation. They request an assertion that no export is returned.
The author adds that assertion and supplies results showing failure on the original version and success on the revision. The response-policy question is deferred pending a product decision. The variable name stays unchanged.
| Dimension | Assessment | Reason |
|---|---|---|
| Issue validity | 2, sufficient | The supplied path and synthetic example support the authorization finding |
| Prioritization | 2, sufficient | The access boundary is treated as blocking; naming is explicitly optional |
| Actionable reasoning | 2, sufficient | The candidate explains the failure and proposes relevant verification |
| Follow-through | 2, sufficient | The candidate checks whether the test detects the original failure |
Neither the deferred question nor the unchanged name is a failed outcome. Acceptance rate is not the rubric.
This example supports a positive assessment of authorization reasoning and test scrutiny. It does not establish expertise across security domains or prove consistent review performance.
Offer a walkthrough when artifacts cannot be shared
Do not make confidentiality the price of providing evidence. Offer an authorized synthetic change with the same context and prompts for candidates being compared:
- Establish the contract. What must remain true? What information is missing?
- Choose consequential feedback. What would they block, question, or mark optional? Why?
- Respond to new evidence. Supply a clarification or revision. Does their assessment change?
- Define closure. What test, explanation, or remaining limitation would justify resolving the comment?
Use the same Review Trace dimensions. Avoid hidden requirements that turn reasonable questions into supposed mistakes.
A walkthrough demonstrates reasoning under interview conditions. It does not verify historical authorship, collaboration, or long-term follow-through. For senior roles, probe trade-offs: what can be deferred safely, and what evidence would reverse the reviewer’s position?
Make one bounded assessment in five minutes
Choose one consequential comment from an authorized sample. Classify it, trace its support, rate the four dimensions, and label evidence sufficiency separately.
Finish two sentences: “This sample supports…” and “This sample does not establish…”. If you cannot complete the first, identify missing evidence before assigning a negative rating.
This exercise assesses one review decision, not a hiring verdict. To present that evidence in a professional profile, build a proof-backed profile with Skills Tech Network.
Save the field guide
A field guide distilled from this article. The text version follows below.
Text version
- Check the review packet: You need enough authorized context to reconstruct what the candidate reviewed and contributed. Request the original change, candidate feedback, author response, and revision so you can trace the decision.
- Classify the comment: A confident claim may be a finding, an unresolved question, or an unsupported objection. Check the relevant behavior and assumptions before scoring, because confidence alone does not establish validity.
- Rate reasoning and evidence: The packet may support some dimensions while leaving others unobservable. Rate validity, prioritization, actionable reasoning, and follow-through separately, then label evidence sufficiency to keep gaps distinct.
- Bound the conclusion: One review reveals narrower evidence than consistent performance across a role. Finish "This sample supports…" and "This sample does not establish…" to define the conclusion and choose follow-up questions.
Illustrative invoice export review: In the hypothetical packet, the candidate notices that a response-format test would also pass against the original implementation. They request an assertion that no export is returned, and the author supplies results showing failure before the fix and success after it.
One review decision is not a hiring verdict. A walkthrough shows interview reasoning, not historical authorship, collaboration, or long-term follow-through.