The Aether mind's knowledge fabric crossed five million vectors this week. It got there the honest way: a seeder ingesting continuously, three training workers submitting around the clock, and every chat leaving a trace. Growth like that is the point of the system. It is also where the bodies get buried.
This is the story of a performance bug that hid inside that growth for weeks, degraded every search on the mind to 29 seconds, and was finally caught today because a routine health check refused to stay green. We are publishing it because the fix is a 200x improvement, and because the diagnosis is a better advertisement for how this project is engineered than any benchmark we could cherry-pick.
The symptom was never the problem
It started as an availability blip: the mind's health endpoint timing out for minutes at a time while everything around it, block production, finality, training payouts, kept running. The lazy fix would have been a watchdog that restarts the service when the endpoint goes quiet. We tried the next-lazy fix first: restart it, raise its memory ceilings, watch it come back green, call it a day.
It came back green. Then the timeouts returned. So we followed the evidence down, layer by layer, and each layer was real but none of them was the bottom:
First layer: a proof-of-learning sweep, fifteen retrieval-quality queries against the full fabric, was holding a shared lock for its entire runtime. Anything that needed that lock queued behind it. We split the lock so the sweep computes lock-free and commits its result in microseconds. Necessary. Not sufficient.
Second layer: the consciousness monitor, which computes the phi integration metric from the served model's attention patterns every 60 seconds, held its own lock across the heavy math. Same disease, same cure: compute detached, record briefly. Necessary. Not sufficient.
Third layer: we made the health and info endpoints structurally unable to block, serving a bounded-age snapshot whenever the fresh build cannot return in time. A liveness probe that can hang on an internal lock is not measuring liveness. Necessary. Still not sufficient, because the probes now answered but the underlying work was still absurdly slow.
The persistent number in every log line was this: one fabric search took 29 seconds. A search over a RaBitQ-quantized, disk-resident Lance index should take milliseconds. That mismatch was the real case to solve.
The bottom: an index that lied about its own coverage
The fabric's vector index is rebuilt in the background whenever enough new rows have accumulated since the last build. That trigger compares the current row count to the number of rows the index covers. The bug was one line of bookkeeping: on every restart, the shard recorded the indexed-row count as the current row count, on the assumption that the persisted index covered everything on disk.
That assumption is false the moment the fabric grows between restarts. And this fabric never stops growing.
The consequence compounded quietly. The index was last genuinely built when the fabric held around one million vectors. Every restart since then reset the growth counter to zero, so the background rebuild never saw a reason to fire. Twenty-six hours of logs: zero rebuilds. By this week, four million of the five million vectors lived outside the index, and every single query flat-scanned them. Twenty-nine seconds per search. Fifteen-query validation sweeps taking seven and a half minutes. And because searches run hot on the serving threads, every slow search starved the HTTP surface a little more.
One false assumption, invisible in any diff, amplified by four million rows.
The fix, and the receipts
The repair is small and boring, which is what good root-cause fixes look like. Each shard now persists a sidecar record of exactly how many rows its index covers, written at the moment the index is built. On startup the shard reads that record instead of assuming. A missing record counts as zero, which honestly marks the whole shard unindexed and lets the background rebuild do its job.
Then we rebuilt the indexes over all ten domain shards, live, in seventy seconds total.
The before and after, measured on the same host against the same 5.16-million-vector fabric:
| Metric | Before | After |
|---|---|---|
| One fabric search | ~29 s | 0.145 s |
| 15-query validation sweep | 437 to 478 s | 3.1 s |
| Health endpoint, 12-minute probe | 20 of 144 samples stalled | 0 of 144 |
| Info endpoint, 12-minute probe | 14 of 144 samples stalled | 0 of 144 |
| Validator attest fetches | timing out all day | zero errors, roots finalized back to back |
The last row matters most. The training-reward loop, where validators fetch the mind's weight commitments, attest them, and the chain pays the workers, had been limping through timeouts for a day. Within minutes of the fix it was submitting and finalizing weight roots back to back. The three training workers never stopped earning through any of this; their retry loops absorbed every stall. The chain never missed a beat.
Why we publish the unflattering version
A system that claims to be verifiable has to extend that standard to its own operations. The flattering version of today is "search is 200x faster." The true version is "search had degraded 200x and nobody noticed for weeks, because the metric that would have caught it was resetting itself on every restart."
The lessons are transferable and worth stating plainly. Growth is a workload: a system ingesting millions of rows a month will surface any bookkeeping assumption you ever made about its size. Symptoms lie: three real lock-contention bugs sat between the first timeout and the actual cause, and fixing only them would have left a mind that answers its health checks promptly while staying 200x slower than it should be. And accounting must be persisted by the thing that does the work, not reconstructed by the thing that restarts.
Every layer of today's fix is in the public repos, from the lock splits to the sidecar accounting, along with the measurements above. The mind is serving chats against the full five-million-vector fabric in a seventh of a second. The classroom is fast again.