The AethersMind has a cognitive architecture borrowed from the Tree of Life: ten Sephirot, each a domain of thought. Keter for meta-learning and goals, Binah for logic, Gevurah for safety and constraint, Hod for language, and so on. For a long time that structure was mostly conceptual, expressed through routing keywords and attention head labels. The real ambition was always different: make the ten Sephirot literal experts in a mixture-of-experts model, each a specialist the router can call on. This post is about finishing that.
The cautionary tale: V6
We tried this once before and it went badly, and the failure is worth keeping in view because it shaped everything we did this time.
V6 took the Sephirot-as-experts idea and implemented it by replacing the base model's attention mechanism with a custom Sephirot-and-sparse-attention design. The reasoning was seductive: if the Sephirot are the cognitive architecture, they should be woven into the deepest layer, attention itself. The result was a model whose cross-entropy was around 16 nats, worse than random. The custom attention tore apart the pretrained Qwen2.5 base capability faster than any amount of training could put it back together. We abandoned V6.
The lesson was blunt and it is now a hard rule: never replace the base attention. The pretrained base is load-bearing. Whatever the Sephirot become, they have to sit around the frozen base, not inside its attention.
V7: the safe but small version
V7 obeyed that rule. It added the ten Sephirot as a mixture of low-rank adapters, LoRA experts, inserted between the frozen base hidden states and the frozen output head, with a learned top-2 router. It worked, and it shipped, and it is the model serving today. But it was deliberately tiny: rank 16 experts, about 1.18 million trainable parameters total. That smallness was a feature for the distributed training economics, but it also meant the experts could only nudge the base, not carry real specialized capacity.
V8 is the step from a small adapter to a real mixture of experts.
What V8 actually is
V8 replaces the low-rank adapters with full feed-forward experts, the same SwiGLU shape the dense model uses, and follows the modern MoE design that DeepSeek-V3 and Qwen3-MoE converged on: fine-grained routed experts plus an always-on shared expert. Ten routed experts, one per Sephirah. One shared expert that every token passes through, which captures the common computation so the routed experts are free to specialize. The block is additive and sits around the frozen base, never touching its attention. The V6 rule holds.
Building a real MoE means solving two problems that a small adapter could ignore.
Problem one: routing without collapse
A mixture of experts is only useful if different tokens actually go to different experts. The failure mode, and it is a very common one, is router collapse: the router discovers early that one or two experts are slightly better and sends everything to them, the rest starve, never train, and you have paid for ten experts to get two. The classic fix is an auxiliary load-balancing loss: add a term to the training objective that punishes imbalance. It works, but it has a real cost. That extra loss term fights the language-modeling objective. You are now optimizing two things at once, and the balancing term injects gradients that have nothing to do with predicting the next token.
DeepSeek-V3 introduced a cleaner answer that V8 uses: auxiliary-loss-free load balancing. Instead of a loss term, each routed expert gets a small bias value that is added to its routing score at selection time only. The biases are not trained by gradients. They are nudged directly by a simple rule: after each step, look at how much load each expert took, raise the bias of the starved experts and lower the bias of the overloaded ones. The router's gradients stay entirely focused on the language objective, while a separate, gradient-free controller keeps the experts balanced. The bias affects which experts are selected, never how much their outputs are weighted, so it steers utilization without distorting the model's actual computation.
We implemented exactly this. There is a per-expert bias vector, an update rule that moves it toward uniform load, and a test that proves a large bias can force a starved expert into the top-k even when its raw affinity is not the highest. The balancing has teeth, and it has them without touching the loss.
Problem two: sparsity
The point of a mixture of experts is that each token only uses a few experts, not all of them. V8 routes top-k: it scores all experts with sigmoid affinities, picks the best k, and renormalizes the gates over just those k so they sum to one. A token engages two of the ten Sephirot, plus the shared expert, not all ten. The current implementation computes all experts and masks the unselected gates to zero, which is mathematically identical to true sparse routing; the further optimization of physically skipping the unselected experts' compute is a FLOP saving we have flagged for later, the same tradeoff the V7 adapter already makes. For a ten-expert block the routing math is what matters first, and it is correct and tested.
Keeping the V6 guard
One detail ties V8 back to the lesson that started this. Each expert's down-projection is initialized to zero. That means at step zero the entire MoE block outputs exactly nothing, so adding it to the residual stream is the identity. The model at initialization is bit-identical to the frozen base alone. Held-out loss at step zero equals base loss. The model cannot be lobotomized at birth the way V6 was, because at birth it simply is the base. Training then grows the experts out from zero. There is a test that asserts this identity property holds.
What is done and what is next
What is done: the V8 SephirotMoE is implemented and tested, behind a feature flag, with top-k sigmoid routing, an always-on shared expert, DeepSeek-V3 auxiliary-loss-free load balancing, the zero-init regression guard, and routing statistics exposed for the consciousness metric the same way the V7 adapter exposes them. The architecture that V6 got wrong and V7 approximated is now built correctly.
What is next, stated honestly: this is the architecture, not yet the served model. Wiring V8 into the live generation path, training the experts on the distributed corpus, and re-measuring against the V7 benchmark standard is the follow-on work, and it pairs naturally with the DiLoCo training loop. The ten experts exist and balance themselves. Teaching them their ten domains is the next run, and we will report it with numbers when it is real.
The Sephirot were always supposed to be experts. Now they are.