The AI-Era Interview Needs a Standardized Customer

Reporting: a new kind of work sample

On Nov. 2, developer Laurie Voss proposed a junior-engineering exercise in which one fixed AI model plays a customer with hidden requirements while a second fixed model builds the software the candidate directs. The candidate asks questions, changes course if needed, and discusses the result with a human interviewer. Voss is explicit that this is a proposal, not a tested hiring method. S1

The useful shift is from “can you produce code on demand?” to “can you discover what should be built?” But an interview like this only compares candidates if the conditions stay stable: same starting brief, same model behavior, same context and time limit, and the same rubric. Otherwise, a helpful simulated customer can become an accidental advantage.

The fairness problem is hidden in the prompt

A Hacker News discussion around the proposal captures a real design tension. One commenter favors a turn-based scenario that reveals information as the candidate asks; another says candidates should be told the topic in advance and given time to research. S2 These are individual opinions, not evidence that either format predicts job performance. But they expose the choice: surprise can test adaptation, while advance notice can test preparation and reduce the advantage of guessing the interviewer’s expectations.

The practical compromise is to disclose the rules and skill being assessed, then keep the underlying scenario consistent. Standardize follow-up branches rather than improvising new obstacles for each person. Let candidates choose their approach, but do not let the interviewer quietly change the difficulty.

Score decisions, not an “AI vibe”

The commercial direction is visible: in its April 2, 2026 launch announcement, CodeSignal described agentic assessments where candidates interpret requirements, build with AI tools, and explain their technical decisions to a human reviewer. That is a vendor’s account of its product, not independent validation of its hiring value. S3 A Y Combinator profile of Magic Patterns engineer Alex Lee also describes Cursor, Claude Code, and an internal code-review agent in his working toolkit—an individual profile, not a hiring study, but a reminder that tool-assisted work is part of some real engineering roles. S4

For hiring teams, publish a one-page test contract: permitted tools, the environment, the task, timebox, and what reviewers will score. Anchor the rubric in observable choices—useful questions, explicit assumptions, sensible scope, and whether the candidate can explain why the result meets the brief. Keep a human responsible for the decision; do not turn a model-generated fluency score into a verdict.

The point is not to stage a perfect miniature of work. It is to make sure hidden variation in the test does not masquerade as a difference in talent.