JetSpec++
Foresight Parallel Drafting with Self Refinements
1UC San Diego 2UIUC 3Nanjing University 4Meta 5StepFun *Equal contribution
Speculative decoding speeds up a large model only as far as its drafts are cheap and its acceptance is high. A strong learned draft model that reads the target's hidden states can draft only after each verification, so its cost stays on the critical path. JetSpec++ keeps the strong draft model and hides almost all of its cost. It drafts the next block on a second GPU while the target verifies the current one. It predicts the verification outcome with foresight drafting, and it refines the draft with Jacobi passes while it waits. With the target and the draft model on two GB200 GPUs, it decodes Qwen3-8B up to 7.45× faster than autoregressive decoding and 1.54× faster than DSpark.
Approach
How JetSpec++ works
Pipelined drafting and verification
Draft-verification pipelining drafts block t+1 while block t is being verified. The draft process anticipates the verification outcome, the number of accepted draft tokens together with the bonus anchor token, and fills a speculation cache with one pre-drafted block per anticipated outcome. A cache hit makes the matching pre-drafted block the next block, and a cache miss falls back to an ordinary speculative step.
Foresight drafting
Foresight drafting consists of an adapter, a decoder and the draft model. The adapter predicts, in one parallel pass, the target hidden states of every position in the block under verification. The decoder reuses frozen weights from the target's last few layers, followed by the target's final norm and LM head, and turns these predictions into the outcome distributions. One decoder pass scores every outcome at once. The draft process selects the most promising outcomes and drafts them in one batch. Without context refresh, the draft model keeps 95 to 98% of the acceptance length it reaches from the target's true states.
Self refinement
The video traces one round of self refinement under verification: a refresh, then a Jacobi pass.
A parallel draft model marginalizes each position over its possible predecessors. JetSpec++ instead lets the draft model refine its own draft in parallel, in the time that pipelining frees under verification.
Self refinement combines three operations. In a Jacobi iteration, each position conditions on the tokens drafted before it. In a context refresh, the adapter re-predicts the target hidden states of the current draft. In an anchor replacement, the decoder finds the first position at which its most likely token differs from the draft and writes its token there. The next pass redrafts only the later positions.
Lossless
Every committed token is verified by the rejection sampling acceptance rule, so the output distribution is unchanged.
Serving
Decode speed on GB200
JetSpec++ runs on its own serving engine with CUDA graphs, with the target on one GPU and the draft model on a second GPU.
With NVFP4 weights and activations, JetSpec++ decodes MATH-500 with Qwen3-8B at 4,117 tok/s on two GB200 GPUs at batch size 1. In bf16, it decodes 1,361 tok/s: 7.45× over AR and 1.51× over DSpark.
Acceptance
Tokens per round on seven benchmarks
JetSpec++ commits more tokens per verification round than every baseline on all seven benchmarks, at temperatures 0 and 1 and at verification budgets 16 and 32. At temperature 0 and budget 16, J3-R3 averages 8.94 tokens per round on Qwen3-8B, against 6.89 for DSpark.
Batching
Throughput and per-user speed
Larger batches raise total throughput but slow down each request. In preliminary runs on Qwen3-8B and MATH-500, JetSpec++ leads DSpark on both throughput and per-user speed at batch sizes 2 to 4.
Next
Coming soon
Larger targets. As target size grows, verification becomes more expensive relative to drafting, which creates a larger window for refinement. We plan to train JetSpec++ for production models that span many GPUs, where the draft model would add only a small share of the hardware.
Higher batch sizes. We are extending JetSpec++ to higher batch sizes and realistic serving workloads. The goal is for drafting to fit around the batched verification schedule, so that the GPUs serving the target model stay fully utilized.
BibTeX
@article{feng2026jetspecpp,
title = {JetSpec++: Foresight Parallel Drafting with Self Refinements},
author = {Feng, Zhaoxiang and Hu, Lanxiang and Yuan, Haoran and Qian, Yu-Yang and Yang, Jihui and Zhao, Peng and Zhu, Yibo and Rosing, Tajana and Zhang, Hao},
journal = {arXiv preprint arXiv:2610.19792},
year = {2026}
}