Running LFM3 on an RTX 4060 Laptop GPU with SGLang

A small language model can be fast on a modest GPU, but a single throughput number does not tell the whole story. In this field report, we describe an October 2026 experiment with Liquid AI's LFM3-1.7B on an 8 GB NVIDIA RTX 4060 Laptop GPU: what broke, what changed, and what one measured workload actually showed.

The headline result: one ramped-arrival run produced 3,647.2 generated tokens per second across the complete run, with 128 requests and a server limit of 64 running requests. This is aggregate throughput across many requests—not the speed of one response, a guarantee of sustained steady-state decode, or a claim about model accuracy. Reasoning tokens are included in generated tokens. Different traffic, input lengths, hardware settings, and software versions will change the result.

The setup

We used a manually quantized FP8 copy of LFM3-1.7B with FP8 KV cache, an RTX 4060 Laptop GPU with 8,188 MiB reported memory, and an SGLang prerelease with native model support. During this experiment the implementation was being developed in SGLang PR #46858; check its current status and the relevant release documentation before reproducing the setup.

The workload used a shared 500-word system prompt, variable tasks, and arrivals spaced 0.1 seconds apart. Thinking was enabled, and each request had a 2,048-token output cap. The machine was operating with elevated GPU clocks during the recorded run; its reported median graphics clock was 2,700 MHz and median power was about 97.9 W. The result is not a stock-settings benchmark.

What we observed

The final three rates come from server counter deltas divided by run duration; the prompt-compute estimate subtracts cached prompt tokens from total prompt tokens. The run recorded 89,854 prompt tokens, including 63,488 cached tokens, or 70.66% cached. The saved duration is rounded, so recalculating from the displayed duration can differ slightly from the run's recorded generation rate. Download the numeric benchmark summary.

These are observations from one run, not a comparison across matched steady-decode windows. Queueing influenced time to first token; a p50 of nearly five seconds is important context alongside the aggregate rate. The benchmark did not establish answer quality across use cases or quantify output quality against an unquantized baseline.

Why the initial output looked broken

Our first SGLang build did not have native support for the model and used a generic Transformers fallback. In our tests that path produced incoherent output, while a separate Transformers control produced coherent output. A manually quantized checkpoint did not by itself fix the fallback behavior. That observation points to a runtime compatibility issue in the tested stack; it does not prove that every version or every fallback path has the same failure.

The native implementation in the prerelease allowed us to run the model with captured CUDA graphs and obtain coherent responses in our spot checks. We also had to align quantization exclusions for the output head and attention gate projection with the runtime's layer matching. Those details are version-specific: consult the current implementation and checkpoint configuration instead of copying a workaround without testing it.

Keep thinking and answer text separate

The chat template in our setup prefilled an opening thinking tag. As a result, a client expecting to see the opening tag in the generated stream could misclassify the output. We configured a server-side reasoning parser and updated the client handling for the prefilled case.

This is more than a display issue for benchmarking. With thinking enabled, generated reasoning consumes output tokens and affects latency and throughput. If a benchmark counts reasoning tokens as generated tokens—as ours does—say so. Do not present the resulting rate as final-answer tokens per second.

How to reproduce responsibly

1. Record the GPU, driver, runtime, model revision, quantization, server arguments, clock and power settings.
2. Verify output coherence separately before optimizing performance; spot checks are not a formal quality evaluation.
3. Record the server's running-request limit, request arrival pattern, prompt sizes, output caps, and cache state.
4. Separate whole-run throughput from matched steady-decode measurements and report latency percentiles.
5. Report fresh prompt computation, cached prompt tokens, and generated tokens separately. Never count cached input as generation.

For the longer working notes, see the original public draft. This article corrects its description of the 3,647 tok/s figure: it is the measured whole-run aggregate, not a single-stream or verified sustained steady-state result.

Need help with local inference or an AI-agent workflow? Swamix Global AI Solutions provides custom implementation and support. Contact Nikhil · swamix.com