The Agent Speedup Hiding in a Shared Context

Batch the questions that share evidence

An agent may read one support ticket, issue, or incident report and need to decide its owner, urgency, category, and next action. Asking a general-purpose model to return JSON works, but it still treats a set of finite choices as a text-generation task. In a Nov. 10 explainer, Hugging Face describes decision models as evaluating predefined answers against a shared state, then returning structured choices and probabilities S2.

That interface is a useful building block, not a replacement for a coding agent. A selector can handle routing, triage, and other bounded branches; a generative model can take over when the work needs explanation, synthesis, or an open-ended plan. The important question is whether several decisions can use the same evidence.

Reuse the context, not just the model

Flide is an inference engine for Jev-style decision models. Its project README describes “exact state forking”: ask several questions about one state without recomputing that shared context for every branch. The README’s own email-triage example reports eight questions for each of 10,000 emails in 28 seconds on one H200, using OneJev1.6-0.8B—a vendor benchmark, not an independent result S1.

The Flide team’s Hacker News launch discussion makes the performance boundary clearer: its “up to 9.5×” result was for Qwen3.5-0.8B on QuALITY, with long articles averaging about 5.8K tokens and 18 questions. The same discussion notes that a one-question request has no shared context to reuse S3. So the headline is not a general-purpose inference multiplier; it describes a workload with substantial repeated evidence.

Make it earn a place in the loop

For a practical trial, collect a real batch of tasks that need multiple decisions from the same input. Put shared evidence in one state, define the allowed answers—including an abstain or human-review route—and compare end-to-end latency and decision accuracy against the existing model path. Measure single-question cases separately; a batch-oriented optimization can disappear when requests do not share context.

Treat returned probabilities as signals to validate, not as a safety policy by themselves. Check calibration on held-out examples and route uncertain or high-impact choices to a stronger model or a person. Then let the general agent do what it is good at: investigate exceptions, explain decisions, and carry out approved work.

The practical takeaway is modest but useful: for agents that repeatedly inspect one artifact, the next speedup may come from asking several well-typed questions in one pass—not from making the model that writes the answer larger.