From sixty seconds to five: hunting the chat latency and shipping TurboQuant P2

Yesterday we wrote about finding a 29-second search hiding inside a growing knowledge fabric. Today's story is the same genre, one layer up: chat on the live Aether 7B felt slow, and on the public site it sometimes did not answer at all. The fix ended up spanning three components that had nothing to do with each other, plus one genuinely satisfying kernel-level root cause.

Here is the honest before and after, measured on the production node through the real public stack, Cloudflare edge included:

Measure first, theorize never

The report was "it still seems slow", which is not a bug ticket you can act on. So the first hour was pure measurement, walking the request path layer by layer.

The cheap endpoints were innocent. Metadata calls answered in under 2 milliseconds locally. Knowledge search, freshly fixed the day before, held at 0.3 to 0.5 seconds with the reranker in the loop. But a one-sentence chat answer took 21 seconds locally on a good run, 66 seconds on a bad one, and through the public site it did something worse than being slow: at exactly 30.5 seconds it returned a bare "Internal Server Error" while the model finished its answer into the void.

That 30-second cliff was the first find. Next.js rewrites, which our frontend uses to proxy /api/* to the API gateway, ship with a 30-second proxy timeout by default. Any generation that ran longer looked like a server crash to the user. One config line, proxyTimeout: 95_000, staying under the Cloudflare edge cap, and slow became merely slow instead of broken.

The second find came from the journal. The mind's log was a wall of Malkuth sealed chat action lines, one every 30 seconds or so. Our background chatter service, which keeps a continuous conversation going against the chain for the public feed, was configured to fire a new chat every 3.3 seconds. Each generation serializes on the model, so a real user's request queued behind whatever the chatter was mid-way through. With generations taking 20 to 60 seconds, the queue wait alone could double the latency. The chatter now runs every 5 minutes. It is colour, not load-bearing traffic, and it should never compete with a person.

The 17x that was hiding in plain sight

Those two fixes made chat fail less and queue less, but the raw number was still absurd: with every background service paused, a 45-word answer took 65.8 seconds. A quantized 7B on an RTX 3080 Ti should decode an order of magnitude faster than that.

We added one log line to the generation loop, splitting time into prefill (priming the KV cache with the prompt) and decode (producing tokens one by one):

[v7-gen] prefill 1590 tok in 1.46s (1092.0 tok/s) | decode 61 tok in 24.37s (2.50 tok/s)

Prefill was healthy. Decode was running at two and a half tokens per second. And one A/B test later we had the culprit cornered: with AETHER_KV_QUANT=tq8 disabled, decode jumped to 29 to 42 tokens per second. Our own KV cache quantization, TurboQuant P1, shipped weeks ago for its 3.8x memory saving, was costing 17x in decode throughput.

The irony is that P1 already contained purpose-built fused CUDA kernels that read the packed quantization codes directly and never materialize the dequantized cache. Those kernels were correct, validated against an oracle, and completely innocent. The problem sat one function earlier, in the encoder that turns each new key and value vector into codes.

That encoder assigned codes by looping over the quantizer's decision boundaries, issuing a compare, a cast, and an add on the GPU for each one. At 8 bits there are 255 boundaries. Times two for keys and values, times 24 quantized layers, that is roughly 37,000 kernel launches per decoded token. Each launch costs microseconds, and microseconds times 37,000 is your 400 milliseconds per token. The math of the quantizer was fine. The shape of the loop was the bug.

TurboQuant P2

The fix is the kind you want every performance fix to be: smaller, simpler, and faster in every dimension at once.

Encoding is now a single broadcast comparison of the input against an on-device boundary tensor, followed by a sum. About five kernel launches per call, independent of bit width, chunked so the broadcast intermediate stays bounded even for prefill-sized inputs. With the encoder out of the way, the fused kernels finally get to do their job on the hot path.

P2 also moves production from 8-bit to 4-bit codes: 16 levels, two codes packed per byte, which was the memory target on the roadmap all along. The packing and unpacking paths already existed and were unit-tested, they had just never run behind a fast encoder.

Live results on the production node, same 1.4 to 1.5k-token RAG prompts:

• old tq8: decode 1.6 to 2.5 tok/s
• quantization off: decode 29 to 42 tok/s
• new tq4 with fused CUDA kernels: decode 30.0 to 30.6 tok/s

Quantized decode now runs at parity with unquantized, while the KV cache pays roughly an eighth of the memory. All ten TurboQuant unit tests pass, including the encode roundtrip and the fused-versus-oracle comparisons, and the on-chain attested model identity is untouched: this is serving-path work, the weights and the manifest root are exactly what they were.

The red herring, for the record

One lead went nowhere and deserves a note so nobody chases it again. The unified model exposes a dtype knob, and full-precision f32 looked like an obvious suspect next to bf16 tensor cores. It is not. The quantized GGUF base always computes activations in f32, so the knob never affected speed, and flipping it to bf16 actually breaks generation with a dtype mismatch when the adapter meets the base. Measured, reverted, documented.

What this took

Three commits: the phase-timing instrumentation, the frontend proxy timeout, and the P2 encoder plus tq4 production config. The instrumentation is arguably the most valuable of the three. The entire diagnosis, from "it seems slow" to a cornered root cause, took one log line and one A/B test. The next person who reports chat latency will get an answer from the journal in thirty seconds.

Every number in this post is reproducible on the live system: open the chat on qbc.network and time it yourself.

Further reading

• KV cache quantization, explained
• Running a 7B LLM on a consumer GPU
• TurboQuant comes to Aether: the P1 groundwork
• Finding the 29-second search: the fabric-side fix