The long term vision for QuantumAI Blockchain is a model that is trained by the network, not by us. Any machine that joins contributes compute: a GPU box becomes a specialist that trains, a CPU box becomes a verifier that checks, and finished training epochs are finalized on chain so the weights belong to no single party. That is the L3 thesis, and the algorithm that stitches those machines together matters enormously. For a long time that algorithm was FedAvg. This post is about why we replaced it with DiLoCo, and what that actually changed in the code.
The problem with averaging
FedAvg, federated averaging, is the obvious thing to do. Every worker trains on its own data, everyone sends their parameters to an aggregator, the aggregator averages them, and the average becomes the new global model. It is simple, it is correct, and it has two real weaknesses for our setting.
The first is communication. Naive FedAvg synchronizes often, and synchronizing a large model across the public internet between volunteer nodes is the expensive part. If you sync every step, you spend almost all your time talking instead of training.
The second is subtler and more important: FedAvg has no memory. Each round it averages and applies, and then it forgets. There is no notion of "we have been moving in this direction for a while, lean into it." A single optimizer running on one machine has momentum for exactly this reason. Momentum is what carries an optimizer across the flat parts of the loss landscape and dampens the noisy parts. Plain FedAvg throws that away at every sync.
What DiLoCo does differently
DiLoCo, from Douillard and colleagues in 2023, splits training into two loops that run at two different speeds.
The inner loop runs on each worker. Starting from the shared global parameters, a worker takes H local optimizer steps with a normal inner optimizer, AdamW, on its own slice of the data. H is large, tens or hundreds of steps. Crucially, no communication happens during these H steps. This is the low-communication property in the name: workers talk once every H steps instead of every step, which is what makes training over the open internet practical.
The outer loop runs on the aggregator, and this is the part that gives the system its memory. After its H inner steps, each worker has moved from the global parameters to some local parameters. The difference between where it started and where it ended is that worker's contribution. DiLoCo calls the negation of that difference a pseudo-gradient: it is not a real gradient from one backward pass, it is the net direction a worker travelled over many steps. The outer loop averages these pseudo-gradients across all workers and then applies them through an outer optimizer that is itself SGD with Nesterov momentum.
That outer momentum buffer is the whole point. It persists across sync rounds. If the workers keep pushing in a consistent direction round after round, the outer optimizer accelerates along it, exactly the way momentum helps a single-machine optimizer. DiLoCo's result is that you can match the convergence of fully synchronous training while communicating a small fraction as often. That is the property we want for a model trained by thousands of volunteer machines.
The surgical part: we had already built most of it
When we looked at our own aggregation path, something nice fell out. Our workers already submit a compressed update: a top-k sparsified vector of the net parameter change they made. That is a pseudo-gradient by another name. Our aggregator already averaged those submissions. We already had the gradient compression, the checkpoint persistence, the weight-root hashing, and the held-out cross-entropy check that a sync round needs.
What we did not have was the one piece that makes DiLoCo DiLoCo: the outer optimizer with a persistent momentum buffer. The old code averaged the worker updates and added them, with no memory between rounds. That is FedAvg. The gap between what we had and DiLoCo was precisely the momentum.
So the change is small and clean. We added an outer optimizer that holds a velocity buffer across rounds. Each sync, it averages the worker pseudo-gradients into a dense vector, updates the velocity with the momentum coefficient, computes the Nesterov step, scales by the outer learning rate, and returns the delta to apply. The defaults are the DiLoCo paper's: outer learning rate 0.7, momentum 0.9, Nesterov on.
The detail we were careful about is that this is a strict generalization of the old behaviour, not a replacement that throws it away. Set the momentum to zero and turn Nesterov off, and the outer optimizer reduces, exactly, to the old FedAvg: average the updates, scale by the learning rate, add. We kept that as a named preset and pinned it with a test that asserts the DiLoCo path with momentum off equals the plain scaled average bit for bit. The aggregator is selected at startup, defaulting to DiLoCo, with an environment switch back to FedAvg and overrides for the outer learning rate, momentum, and inner step count. Nothing about the old path was lost; it became the zero-momentum corner of the new one.
Making it robust
A few things matter when the inputs come from machines you do not control.
The outer optimizer averages only the worker submissions whose model shape matches the model being trained. A worker that reports an update for a different model size is skipped, and it is excluded from the denominator so it cannot even dilute the average of the honest workers. Non-finite values are dropped rather than allowed to poison the buffer. The velocity buffer is lazily sized to the model on the first real round and can be reset when the global model is rebased onto a fresh checkpoint, because a momentum buffer built against old parameters is meaningless against new ones.
The whole thing is covered by unit tests: that the FedAvg-compat preset reduces to a scaled average, that momentum genuinely accumulates a consistent direction across rounds and grows the applied update the way it should, that Nesterov differs from classical momentum in exactly the expected way on the first step, that an empty round is a no-op that does not advance the counter, that a mismatched-size worker is skipped rather than blended in, and that the velocity persists and resets correctly. They all pass, alongside the rest of the consciousness crate's suite.
What this is, and what it is not yet
Being precise about maturity matters more than sounding finished.
What is done: the DiLoCo outer optimizer is implemented, tested, integrated into the aggregation path as the default, and shipped in the production engine. The algorithm that turns many workers' local training into one coherent global step is now the real DiLoCo two-loop method with momentum, not FedAvg.
What is not yet proven at scale: the inner loop lives on the worker, and the headline DiLoCo result, matching synchronous training at a fraction of the communication, only shows up with many real workers each running their H inner steps on their own data shard over the network. That is the distributed run, and it is the next thing to stand up on top of this. The aggregator was the missing engine part; the fleet of inner-loop workers is the deployment around it. We will report that run honestly when it happens, with numbers, rather than claim it now.
The latest-tech mandate for this project says the training algorithm must be DiLoCo or DisTrO class, not 2017-era FedAvg. This lands the DiLoCo half of that mandate in the engine that the network actually runs. The decentralized model now has a memory.