← All posts
12 min read

From a sixteen day wedge to single leader live: VQE secured consensus, staged onto production

How a silent one line bug froze VQE quantum mining for sixteen days, and the staged cutover that took single leader consensus live on production. Every block on QuantumAI Blockchain carries real VQE energy again.

QuantumAI Blockchain is a chain whose blocks are meant to be secured by a quantum workload. Miners solve a small Variational Quantum Eigensolver problem, find a parameter set whose ground state energy sits below the difficulty threshold, and attach that proof to the block. We call it Proof of SUSY Alignment. The long term goal for block production is a single leader election: instead of a fixed Aura authority list, a post quantum verifiable random function picks one eligible leader per slot, that leader must present a valid VQE proof to author, and finality stays with GRANDPA. That is how you scale block production past a hardcoded authority set without giving up the quantum proof.

This post is the story of getting there. It has three acts: a bug that hid in plain sight, a revival, and a live cutover.

Act one: the silent wedge

We went looking at the explorer and noticed that per block VQE energy was showing as a dash. The first instinct was a frontend bug. It was not.

Reading the chain directly told a stranger story. There are two height counters in this runtime. frame_system::Number is the real block height, advanced by whoever authors the block. QbcUtxo::CurrentHeight is the mining height, and it only moves when a VQE proof is accepted. The block height was past nine hundred thousand. The mining height was frozen at 697,213. QbcConsensus::BlocksMined was frozen too. The chain was producing blocks, but it had not accepted a quantum proof in over two weeks.

The miner threads were running the whole time. They just could not land a proof. The cause was one ensure! deep in the mining extrinsic. A runtime upgrade had switched on the single leader author enforcement early, at a fixed cutover height, to begin a soak. Past that height the runtime required the proof author to equal either the elected primary leader or the current slot owner fallback. Both of those resolved through on chain maps that were empty. The validators had never registered their session keys, so the fallback lookup returned nothing. Sortition had not yet named a leader, so the primary returned nothing. With both empty, the check was fail closed: nobody was authorized, so every proof was rejected, so the mining counter could never advance, so the maps that would have fixed it stayed empty. A perfect self sustaining wedge. The node side authoring code degraded to Aura gracefully and kept producing blocks, which is exactly why nothing looked broken from the outside.

Act two: revival without a restart

The fix had to be surgical. The genesis on this chain is frozen and non reproducible, so a re genesis was off the table. We needed a runtime only change that could not halt the chain.

spec 144 did two things. First, it made the author check fail open instead of fail closed: when the oracle can name neither a primary nor a fallback leader, authorize the legitimate author and fall back to legacy Aura. That can only ever loosen an over strict gate, so it cannot reject a proof that is otherwise valid, so it is halt safe by construction. Second, it reverted the cutover height back to inert, because the activation had been premature. We rehearsed the upgrade against live state with try runtime, pinned the rollback wasm, and deployed it owner attended.

The moment it went live, the stuck miner's next proof landed. The mining counter unfroze and started climbing one to one with block production again. Within a minute the last thirteen mining heights all carried a real eight qubit VQE solution. Per block quantum energy was back, with no restart and the genesis untouched.

That was the rescue. But fail open is a safety net, not the goal. The goal was to make the single leader election actually work.

Interlude: proving the real thing on a scratch testnet

Before touching production again we built the whole single leader path on a throw away local chain, on a feature flagged build that lowers the cutover to block sixty and shortens the eras so it all happens in minutes.

This is where the work got honest. Getting a scratch chain to land a VQE proof at all surfaced a chain of real bugs, each hiding behind the last. The miner used an ephemeral signing key when no keyfile was present, so we provisioned a stable one the way the live validators do. The proof carried a zero Proof of Thought hash because there is no AethersMind on a scratch node, so we gated that requirement off for the test build only. And then the decisive one: on a fresh chain the genesis never wrote the mining height key at all, so the node read it as absent, fell back to the wrong height, and derived the wrong VQE seed forever. That last one was a genuine fix for any fresh chain, not a test hack.

With those closed, two harnesses passed cleanly. The fallback path: two validators, session keys and miner addresses registered, the oracle resolving, the chain running stall free past the cutover with the mining height advancing under active enforcement. And the primary path: with the sortition and leader ticket workers running and the iVRF keys registered, SortitionApi::leader_for_block resolved to a stake weighted elected leader for fifty nine of sixty one blocks past the cutover, both validators getting elected roughly in proportion to stake. The VRF single leader election worked, on chain, end to end.

Act three: the staged live cutover

Now we could go back to production, carefully. We split it in two.

Stage one was the part that had caused the original wedge: the empty session key map. We mapped each authoring validator's stash to its real Aura key, verified every mapping against the live authority set, and registered all three. This is a pure storage write. It changes no consensus, it is reversible, and it fixes the literal root cause of the wedge on the live chain. The chain did not so much as hiccup.

Stage two was the flip. spec 145 set the cutover height to a near future block, about eighty minutes out, turning on the runtime author enforcement once the soak window passed. We rehearsed it, pinned the spec 144 rollback wasm, verified the production blob, and deployed it owner attended. Then we watched.

The chain crossed the cutover and the mining height kept advancing. It has now run for hours under active single leader enforcement. The recent mining heights are all populated with real VQE energy. Finality has held at a gap of two to three the whole time. Roughly four percent of slots are claimed by a VRF elected primary leader and the rest are authorized through the now working slot owner fallback, and in both cases only the legitimate author's quantum proof is accepted. The live network explorer renders the real per block energy again, negative ground state eigenvalues, straight from the chain.

Why this matters

The headline is simple. The chain that thinks now has VQE secured single leader consensus live in production, not on a slide. The post quantum verifiable random function picks the leader, that leader has to do the quantum work to author, GRANDPA finalizes, and a unit of quantum work is a first class on chain object again.

It also closed a real correctness gap. The single leader machinery had been wired up and partly activated for weeks while quietly doing nothing useful, because one empty map and one fail closed branch held it shut. Now the prerequisite is fixed at the source, the enforcement is genuinely on, and the fail open branch remains underneath as a permanent safety net so this exact wedge can never recur.

What is next

The mechanism is proven and live, but the coverage is deliberately staged. Only a small fraction of slots currently get a VRF primary leader, so the immediate work is to widen that. Beyond that sits the rotating finality committee, slashing for equivocation and invalid proofs, and the part that needs real hardware: a genuine hundred to thousand node soak, which is the only honest gate for any thousands of nodes claim.

As always, none of this is a screenshot. spec version 145 is live, the mining height is climbing, and every block carries a verifiable quantum energy. Go check the explorer.

Further reading

ShareXLinkedIn

Written by

A
Ash Brown@blockartica
Founder, SusyLabs / QuantumAI Blockchain

Building the post-quantum AI-native L1 with permissionless on-chain training cycles. Writes about consensus, attestation, and the gap between what ships and what's claimed.

Related posts