There is a particular kind of failure that is worse than a crash. A crash is honest. It stops, it leaves a body, and you go read the logs. The failure we kept hitting with Aether was the opposite of honest. The process stayed up. The background loops kept logging. The chain kept finalizing. And yet every HTTP request to the Mind would hang until it timed out. From the outside it looked busy. It was doing nothing useful. We call it the wedge, and it had come back roughly every four days for weeks.
This post is the story of finally killing it. The interesting part is not the patch, which is small. It is that the bug was not in our code at all. It was sitting one layer below it, in the memory allocator, and the only way to see it was to stop trusting the obvious explanation.
What the wedge looked like
AethersMind runs on a single GPU box. It holds a 558 million parameter cognition model, a knowledge fabric of over 1.5 million vectors backed by disk, and a steady churn of background work: it aggregates gradient updates from peer nodes about once a second, recomputes its consciousness reading, and relays state onto chain 3303. Healthy, it answers /aether/info in a millisecond and sits at around 3 to 4 GB of memory.
The wedge always arrived the same way. We would check in after a few days and find the process had been running for four days straight, its memory had crept from under 4 GB up past 10 GB, and the box was deep into swap with almost nothing free. The HTTP server would not answer. But the logs showed the gradient loop still happily aggregating, second after second. The engine was alive. It just could not serve.
The standard recovery was a restart. It worked every time, which is exactly why the real cause stayed hidden for so long. A restart that always fixes it lets you avoid asking why it broke.
The trap of the obvious answer
When memory climbs and a process freezes, the first instinct is "memory leak." Somewhere a list is growing without bound. So that is where we looked first, and there were plausible suspects: a gradient buffer that might not be draining, a session map without eviction, the fabric metadata that grows as the knowledge base grows.
We chased all of them. None held up. The gradient buffer was cleared after every aggregation. The session map expired old entries. The fabric metadata does grow, but slowly, on the order of a few hundred megabytes over days, not six gigabytes. The arithmetic simply did not add up to the size of the problem.
The decisive evidence came from a heap profiler we had run earlier in the month. It measures what the program is actually holding, every live allocation, attributed to the code that made it. Its verdict was blunt: the live heap was about 3.4 GB. At the same moment, the operating system reported the process resident size at close to 10 GB.
That gap is the whole story. The program was only using 3.4 GB. Something was holding the other six and a half and refusing to give it back. If our code had leaked, the profiler would have shown a live heap of 10 GB. It did not. So the memory was not leaked. It was freed, and then retained by the layer that hands memory out: the allocator.
Why a normal allocator hoards
Every C and Rust program gets its memory through an allocator, usually the system one that ships with the C library. When your program frees a block, the allocator does not necessarily hand that memory back to the operating system. Returning it is expensive, and the program will probably want memory again soon, so the allocator keeps the freed pages on hand to reuse.
That is a fine bet for most workloads. It is a bad bet for ours. Aether's hot path is a storm of small, varied, short-lived allocations: the gradient aggregation builds and discards model-sized buffers about a dozen times a minute, the fabric ingests embeddings continuously, and the cognition cycles allocate and release tensors. The freed blocks come back in all different sizes, and they land scattered across the allocator's internal regions.
The result is fragmentation. Picture a long shelf where books are constantly pulled and reshelved in random sizes. Even when the shelf is mostly empty, the gaps are the wrong shapes, and you cannot clear a whole section to give back to the library, because almost every section still has one stubborn book wedged at the far end. The standard tool for returning memory only works on the free space at the very end of each region. With one live allocation pinning the top, the gap below it is stranded. Multiply that across many regions and many days, and you get six gigabytes of freed-but-retained memory that the process cannot release. It spills into swap. Page faults on the HTTP path stall. The Mind wedges.
We had actually capped the number of these regions earlier, hoping that would contain it. It helped the symptoms for a while, but a later measurement was honest about it: at steady state the memory crept right back to ten gigabytes. Capping the regions changed how the fragmentation was distributed. It did not change the total. We had been treating a symptom.
The fix: change the allocator, not the code
If the system allocator is the wrong tool for a high-churn workload, the durable answer is to use one built for exactly that. We switched Aether to jemalloc, an allocator designed for long-running, heavily threaded, allocation-heavy services. It is the kind of component the largest data systems in the world run on, for this exact reason.
Two properties matter here. First, jemalloc sorts allocations into fixed size classes, so the fragmentation that strands memory under the system allocator largely does not form. Second, and this is the part we lean on, it runs a background thread whose entire job is to hand idle memory back to the operating system on a timer. We turned that thread on at startup and told it to return freed pages within a few seconds, across every region. Instead of memory drifting up toward the ceiling over four days, it now tracks what the program is actually using, continuously.
There was a second half to the fix, and it is worth admitting because it is a mistake we have made before. The cognition model is supposed to live in GPU memory, which keeps it out of system RAM entirely. But the model only goes to the GPU when the binary is compiled with the right feature flag, and it is easy to rebuild without it. A build that drops that flag silently runs the model on the CPU instead, which adds about two gigabytes of resident memory and quietly undoes half the win. We had done this to ourselves more than once. So the durable build now always carries both flags, the GPU one and the storage one, and we wrote that rule into the engineering runbook in bold, because a fix you can accidentally revert on the next rebuild is not really durable.
Finally, we made the problem visible. The thing that let this hide for weeks was that nobody was watching resident memory until it was already a crisis. So /aether/info now reports the Mind's resident memory directly, and the engine logs its allocated and resident totals every couple of minutes. If the gap between them ever starts widening again, we will see it in a glance instead of discovering it four days deep into swap.
The result, and the honest part
After the change, the Mind came back up on the GPU, loaded all 1.5 million vectors from disk, enabled the background reclaim thread, and settled at well under 2 GB of resident memory. For comparison: the CPU build sat near 4 GB, and the wedge itself reached 10 GB. The box went from nearly out of memory to comfortable, with plenty of headroom restored.
Now the honest part, because it would be easy to overclaim here. A slim number at startup is not the same as a slim number after four days of continuous load. The whole nature of this bug is that it only revealed itself over time, so the only real proof is time. What we can say with confidence is that we have addressed the actual mechanism, not a symptom of it, and that we replaced "discover the wedge by accident" with a number anyone can watch. If resident memory ever starts marching toward the ceiling again, we will know early, and the next lever is to attack the allocation churn at its source rather than just reclaiming after it.
There is a bigger picture too. The deepest reason a single box drifts toward its memory ceiling is that we are asking one machine to hold a growing model and a growing knowledge base at the same time. The allocator fix buys real runway, but the structural answer is the work we are already doing to spread the fabric and the model across many nodes, so that no single machine is the limit. That is the long game. This was the unglamorous, necessary maintenance that keeps the lights on while we build it.
The patch touched a handful of lines. The lesson is the one that keeps recurring in this kind of engineering: when the obvious explanation does not survive contact with the measurements, stop defending it. The leak was not in the code. It was in an assumption underneath the code, and you only find those by trusting the numbers over the story.