Score the loop, not the label

A coding agent is more than a model with a chat box. Its tools, prompts, control flow and task distribution shape what the model actually does. Improving the checkpoint alone may miss the bottleneck—or optimize behavior for a setup your team does not use.

In a November 20 report, Hugging Face trained LoRA adapters for Qwen3.8-27B inside three agent harnesses: pi, Hermes Agent and OpenCode. In its Terminal-Bench 2.1 example with pi, the base model scored about 39% and used an average of 2.9K tokens per task; a custom adapter raised the reported score to 55% and cut average use to 1.5K. But the report’s 300-task AutomationBench check was less conclusive: adapted variants scored 17.7% to 27.0%, against 23.0% for the base model, and the authors describe the differences as within noise except perhaps for Hermes. These are the publishers’ experiments, not a promise that fine-tuning transfers to your workload. S1

The engineering takeaway is to benchmark the combination you intend to ship. Hold the model, harness version, tool schema, prompt and task set steady; then compare completion rate, tool calls and token cost. A better model score on a different loop is not evidence of a better agent for your team.

Memory is part of that runtime

The same logic applies across sessions. Persistent memory can help an agent avoid relearning a verified convention, but stale or indiscriminate notes can become another source of error and context cost.

Anthropic’s November 6 engineering post describes a background “dreaming” process that reviews session records to correct notes and organize memory. In one evaluation, 128 sessions built a wiki over an internal document corpus. On a 25-question set, wiki-plus-search performance rose from 54% to 61% as the wiki grew, while search alone scored 53%. That is a narrow, company-run result, but it suggests a practical design: retain useful knowledge, preserve its evidence, and let the agent check it against the record instead of treating every old note as truth. S2

Put changes behind a boundary test

For a team adopting an agent, start with a small, representative suite of real tasks. Record success, regressions, tokens, tool calls and whether the agent respected tool boundaries. Keep a baseline when changing a model, harness, prompt or memory policy, and rerun that same suite after tool or workflow changes.

Keep session logs searchable, but promote only durable, verified conventions into always-loaded context; refresh or remove notes when the underlying tools change. The useful unit of evaluation is the whole loop—from request, through tool use and retained context, to a checked result. A local benchmark gain is a reason to test the change in your workflow, not proof it will improve developer productivity there.

Sources and editorial method

This AI-assisted report synthesizes the sources below. Automated source checking is not independent human verification; follow the original reporting for context and updates.

Common questions

What should an AI coding-agent evaluation hold constant?

Keep the model, harness, tool schema, prompts and task set fixed when comparing changes; report success alongside token use and tool calls.