Residual Reuse Looked Promising but Made Decoding Slower

The original residual reuse idea

Normal decoding processes the current token through every layer, produces logits for the next token, and stores the state needed for later steps. My first question was whether the final residual from that work already pointed far enough toward the following token that I could use it to skip part of the next pass.

I started with Qwen3.5-4B. For a simple prompt, the normal model chose " Paris" as the first token and "." as the next one. I captured the final residual before normalization after the model chose Paris, then tried several ways of feeding that information back into the model.

Raw reuse was terrible. The residual had an RMS of 1.34656 while the token embedding had an RMS of only 0.010845. Subtracting one unscaled embedding from the residual barely changed it. Once I matched the RMS scale, the model recovered the right next token on two simple prompts where a word was followed by punctuation.

Then it failed on a numeric prompt where the first generated token was only a space. The final residual had useful information about the current context, but I could not treat it like a first layer input for the next position.

Matching the residual to the layer helped, but not enough

Feeding a final layer vector back into an early layer was a bad mismatch, so I tried matching the reuse point to the layer instead. I estimated the cumulative update from a skipped prefix, combined a scaled average from the previous two positions with the new token embedding, and ran only the remaining contiguous suffix of the model.

I found five candidates that improved mean Jensen-Shannon divergence across 24 prompts compared with controls that used the same amount of model depth. The best candidate skipped 8 of 32 layers. Across discovery and a separate confirmation set, the selected candidates only matched the exact second token 33.3% to 37.5% of the time.

There was something in the residual worth reusing, but not enough to draft reliably.

The middle of the model had a surprising amount of slack

Next I tested which parts of the model I could approximate or skip. I ran 5,640 combinations of MLP and token mixer changes across different layer ranges.

Changing only layers 9 through 16 was much more forgiving than doing the same thing everywhere. Reusing the mean of the previous two mixer outputs in that window kept the correct next token on all 24 prompts and had a mean Jensen-Shannon divergence of 0.0055. Reusing half of the previous MLP output also kept the same top token every time, with mean Jensen-Shannon divergence of 0.0123.

This only worked locally. Extending mixer reuse to layers 2 through 30 still kept the same next token 91.7% of the time, but layers 9 through 31 fell to 54.2%. Replacing every interior mixer fell to 20.8%, and removing all mixers gave the target token a median rank of 25,896. Some sections had slack. Applying the same trick everywhere destroyed the result.

Skipping complete blocks was more useful than skipping pieces

On Qwen3.5-9B I tested each component and ranked the least damaging changes. DeltaNet mixers in layers 26, 27, 29, and 30 were especially forgiving. When I combined the results, skipping complete blocks worked better than piling up separate MLP and mixer changes.

The speed measurements confirmed that I was really bypassing the blocks. They did not include the cost of running the draft and verifier one after the other. That cost killed the idea. The MAC proxy in the table is my estimate of the multiply accumulate work removed, not a measured reduction in complete decoding time.

Alternating reuse was interesting but still unstable

I also alternated token mixer reuse across continuations of eight tokens. The best version matched 81.2% of teacher forced tokens and 53.1% of aligned tokens when it had to use its own output. It got all eight tokens right on 29.2% of prompts, with a mean exact prefix of 3.79 out of 8.

The difference between those two tests mattered. One early mistake changes the context for every token after it. The method looked pretty good when every step received the correct prior token, then got much worse when it had to consume its own output.

I also tried skipping MLPs in even layers and DeltaNet mixers in odd layers. That removed an estimated 40.5% of token computation. The best version accepted 1.125 of two proposals on average and accepted both 45.8% of the time. Interesting, but still not useful for speed.

Simple partial MLPs beat my clever selectors

I tested subsets of the SwiGLU intermediate dimension in layers 2 through 31 while keeping every token mixer normal. Simply keeping the first part of the intermediate dimension beat selecting neurons by entropy, reverse entropy, L1, residual channels, or energy bands.

Keeping the first 25% looked a lot better when I stopped demanding the exact argmax and inspected the top 20 choices. At an estimated 62.5% of normal token cost, it shared 13.42 of 20 token identities, chose the same top token 75% of the time, and retained 94.9% of the normal conditional probability mass among the top 20 at temperature 1.

Unfortunately, those cost numbers were arithmetic estimates. The experiment used index_select, not a tuned kernel for the smaller network. The outputs looked good, but the implementation was not fast.

Getting rollback right

Qwen3.5 alternates Gated DeltaNet mixers with periodic full attention layers. Attention KV caches can be cropped after a rejected draft, but the stock DeltaNet chunk path only exposed its recurrent state after the final position. That is not enough when the verifier accepts the first draft token and rejects the second one.

I changed the verifier path to record the DeltaNet state after every proposal position. After verification, I commit the exact target state at the accepted boundary and throw away the rejected suffix. I do not repair approximate state or replay the accepted tokens.

Each DeltaNet layer in the tested 4B and 9B configurations used recurrent state shaped as [batch, 32, 128, 128]. At FP32 that is 2 MiB per sequence per layer, or roughly 48 MiB across 24 DeltaNet layers before convolution and attention caches. Recording each proposal boundary temporarily multiplies that part of memory use by the number of positions.

The correct implementation was still MUCH slower

The larger sweep tested 7,680 candidate distributions and 384 exact speculative runs. Keeping 75% of each interior MLP had an expected acceptance rate of about 79.2% at temperature 0.7. With two sparse proposals in sequence, the first draft was accepted 74.6% of the time and the second was accepted 79.6% of the time when it was reached.

Correct rollback removed the need to recompute accepted context. It still only reached 7.626 tok/s against 12.764 tok/s for normal cached decoding. The partial draft path consumed 62.3% of total decode time, verification consumed 31.9%, and bookkeeping used the remaining 5.9%.

I reduced full model calls by more than half and made the whole thing about 40% slower. The drafts were accurate enough. They were not remotely cheap enough.

Versions with four drafts and skipped blocks were not any better. One setup with a top 5 gate produced 3.593 tokens per verifier call and reduced target calls by 72.17%, but only reached 0.675x normal speed. The top 5 gate was intentionally lossy, so it did not preserve the sampling distribution the way exact speculative acceptance does.

Free drafts finally made the verifier worthwhile

I needed to know whether verifying several positions at once and rolling the state back had any chance of producing a speedup, or whether the entire approach was broken. I removed the neural draft path and used an offline n-gram table generated by Qwen to propose phrases of up to four tokens with basically no model computation.

Only 24.28% of the attempted phrase tokens were used, and 59.11% of phrase cycles missed on the first token. It still won because the draft was nearly free and the GPU could check several positions while loading the target weights once.

The full model processed 2.40x as many positions, so this was not more compute efficient. It was faster though. That told me the verifier and rollback code could beat normal decoding if I could make the drafts cheap enough.

Where I ended up

Raw residual reuse did not generalize. Matching reuse to the layer was better but still too inaccurate. I found blocks and MLP sections that could be skipped, but running those partial neural paths cost more than the verifier calls they saved.

The work was not a complete dead end. I got exact rollback working, found which parts of Qwen3.5 tolerated removal, and got a real speedup once I replaced the neural draft with a free n-gram lookup. A useful drafter has to be MUCH cheaper, not just somewhat smaller than the target.

The 24 prompt tables here were for finding things worth testing, not making broad model quality claims. I only treat the wall clock numbers as comparable when they came from the same model, device, runtime, and benchmark setup.

Update, September 1, 2026: After this I moved on to using the stock trained MTP heads with cheaper output projections. That work became Shortlist MTP Mostly Worked Around a Bad Kernel Choice.