Two Optimizations That Were Never Allowed to Run Together
A stock inference server made them choose between fast first responses and fast streaming. Forking it removed the choice — and a re-run exposed a benchmark that had flattered the result.
Summary of a report from Launch80, the NSS research lab. Read the full report →
The stock mlx-vlm server offers two accelerations — speculative decoding for fast streaming, prefix caching for a short wait before the first word — and lets you run only one. Launch80 forked it to run both.
Three changes
- Removed the false choice. The exclusion was a one-line precaution upstream, not an architectural limit.
- Made the model's own output reusable, checkpointing its state at the end of each turn so the conversation is not re-read and a restart comes back warm.
- Rewrote the scheduler so new requests interleave with work in flight instead of queueing behind it.
The result, honestly measured
Against the configuration they had been running, an 18-turn agent session dropped from 141 seconds to 38. With four agents at once: 1.6× the throughput at about a third of the wait. Against stock caching alone the single-session gain is a more modest 1.4×.
The more useful lesson is the one that cost them: their first published benchmark was about three times too optimistic. The fork and the baseline had been measured differently, and the baselines were recorded in Low Power Mode. Someone re-derived the numbers from the raw data and caught it. Their advice: have someone try to break a benchmark before you publish it.
Read the full report on Launch80
Methodology, every table and chart, and the caveats this summary leaves out.
launch80.com/blog/two-optimizations-never-allowed-together →