AI Coding Tests Need a Rulebook, Not a Guessing Game

An “AI allowed” notice is not a useful policy. It leaves candidates guessing whether an assistant can offer syntax hints, write code, or act more freely—and whether their prompts become evidence in the hiring decision. Those differences change what the test measures.

One label can describe different tests

HackerRank’s employer documentation offers three settings: no assistant, guarded assistance, and unguarded assistance. The employer can configure the setting for a test section S1. Its candidate guidance says the hiring company chooses the mode and that candidates are told before the test when the assistant is enabled S2. These are HackerRank-specific capabilities, not evidence that every employer uses the same options.

Still, they expose a general design problem: “AI allowed” does not say what candidates may do, what the tool can do, or how the work will be assessed. A closed-book task and an agent-enabled exercise can probe different skills. Treating their results as interchangeable makes the score hard to interpret.

Assistance and scoring are separate choices

HackerRank also documents an optional AI Fluency Evaluation. It analyzes assistant interactions and IDE activity, then assigns A, B, C, or NA grades. The feature requires Advanced Evaluation and Proctor Mode; the documentation says an NA grade means there was not enough interaction to assess fluency, not that the candidate performed poorly or failed. It also says the grade should complement other evaluation metrics S3.

That distinction belongs in the invitation, not in a surprise at the end: enabling an assistant is one test condition; grading how a candidate uses it is an additional assessment choice. A vendor’s rubric may help organize evidence, but a grade is not self-validating proof of job performance. Reviewers should know what is recorded, what is scored, and how much weight it carries.

Keep the policy fixed; make the task relevant

A November 2 Hacker News discussion about junior-engineer hiring shows the transparency question in practice. One commenter argues that candidates should know in advance what they will discuss and have time to prepare; other comments discuss scenario-based, changing constraints S4. That is a debate, not evidence that one interview format produces better hires.

A practical rule is to keep the conditions consistent while making the work job-related. Tell every candidate which assistant mode is available, whether outside tools are permitted, what interaction data reviewers can see, and whether AI fluency is scored. If unaided fundamentals matter, assess them separately and label that part clearly. If AI collaboration is part of the role, give candidates equal access and evaluate how they verify output and make technical decisions—not simply how often they prompt.