Speculative decoding on RDNA

01 Why decode leaves room

Generating one token means reading every active weight once and doing very little math with each byte. That is a bandwidth-bound job: the GPU's matrix units wait on memory. If you hand the same pass several positions at once, the weights are read once and used several times, so a few extra rows cost much less than a few extra tokens would.

Speculative decoding spends that slack. A cheap draft proposes the next k tokens. The full target model scores the current position plus all k guesses in one verify pass of k + 1 rows. Every guess the target agrees with is a token you got without its own decode step.

  emitted so far:  … the  quick
  draft head:              brown   fox   jumps        (k = 3 guesses)
  verify pass:     row 0   row 1   row 2   row 3      (one target pass, 4 rows)
  target says:     brown   fox     leaps   ·
                   ✓       ✓       ✗ → emit "leaps"
  window result:   brown fox leaps     3 tokens for one verify pass

The window always emits at least one token (the target's own choice at the first disagreement, or a bonus token after the last row when every draft is accepted), so a window is never worse than one decode step in tokens. Whether it is better in time depends on what the extra rows cost. That is the RDNA-specific part (§ 05).

02 The draft is part of the model

Flash-Next carries a native MTP layer: an extra prediction head that reads the trunk's hidden state and proposes the tokens after the next one. hipfire attaches it as the drafter, so there is no second model to download, load or keep in sync. On the two release cards, Strix Halo (gfx1151) and the R9700 (gfx1201), native MTP is on by default; the draft depth defaults to 3 (speculation.mtp_k).

Greedy requests accept the longest prefix of drafts that equal the target's argmax. Without n-gram takeovers (§ 06), 0.4.1's greedy MTP and plain AR agree token for token across the repaired paths: 2–8-row verify projections now sum in exactly the single-row decode order (the gemv_bf16_xf32_k4_rows_r2..r8 kernels), and the Q8 recurrent state is quantized at every verify row the way AR quantizes it after every token.

03 Sampled requests: rejection sampling

With temperature above zero there is no single “right” token to compare against, so agreement has to be defined on distributions. Speculative rejection sampling (Leviathan et al.; Chen et al.) does this without changing what you sample from:

1. Draw the draft token x from the draft distribution q.
2. Accept it with probability min(1, p(x) / q(x)), where p is the target's distribution at that row.
3. On rejection, emit a draw from the residual (p − q)+, renormalized, and end the window.
4. If every draft is accepted, draw the bonus token from p at the last row.

The emitted tokens follow p exactly, the same distribution plain sampling would use. Drafts only change how many tokens one verify pass produces.

p is exactly the Flash-Next AR sampler's distribution: temperature, the request's top-k (absent means 20 candidates, 0 means the 64-wide pool), min-p and top-p, with the same pool, tie order and summation. q applies the same policy to the draft head's 8 re-scored candidates and their exact logits. One request-seeded random stream supplies every draw, so the same seed replays the same tokens.

04 Penalties on every verify row

Repeat, presence and frequency penalties depend on history: which tokens were already produced. In a speculative window that history grows row by row, so each verify row must be penalized against the history as if the earlier drafts had been accepted:

  H_i = last W tokens of ( prompt ‖ emitted so far ‖ drafts[0 .. i) )

  row 0 penalized by  … prompt, emitted
  row 1 penalized by  … prompt, emitted, d0
  row 2 penalized by  … prompt, emitted, d0, d1
  W = the request's penalty window; a rejected draft never enters the next window's history

Both p (the target row) and q (the draft step proposed for that row) are penalized by the same H_i, so rejection sampling still targets the distribution a penalized AR sampler would use. Target rows are penalized on the GPU by a prepass that is bit-identical to the host penalty stage (same emitted IDs on 24 of 24 prompts on each architecture), so only the already-penalized row is downloaded.

In 0.4.0, native MTP verification silently dropped these penalties. Agent workloads often send them (the release's agent evaluations sample with a presence penalty of 1.5), so penalized requests now speculate with the distribution they asked for.

05 What a verify row costs on RDNA

“Extra rows are cheap” is only true up to a point, and the point is different on every card. hipfire 0.4.1 carries measured window-cost fits for Flash-Next on both release cards and uses them to price decisions:

One row (a plain decode step) costs about the same on both cards. Each extra row adds 6.9 ms on Strix Halo but 14.8 ms on the R9700. Divide a window's cost by one decode step and you get the number of tokens the window must emit just to break even with AR:

Ratios computed from the fits above (window cost ÷ one-row cost). Up to 8 rows the fits are explicit per-row buckets; 9 or more rows run a different expert-sharing route, which 0.4.1 does not extrapolate to and does not assume is cheap.

This matches the gap between the two cards in § 08: the same head gains far more where extra rows are cheap. It is also why hipfire chooses the draft depth per window: it picks the depth k that maximizes expected emitted tokens (1 plus the running product of per-depth acceptance) per unit of window cost, and falls back to one-token-per-step when no batched window beats it.

06 N-gram takeovers, priced on the same clock

Agent and coding traffic repeats itself: file paths, identifiers, JSON keys, quoted tool output. A request-local n-gram pool, seeded from the full prompt, can propose those continuations for free. In 0.4.1 such a match may take over a window from the MTP head (default triple 5/3/3: match 5 tokens, propose 3).

A takeover only happens when its expected emitted tokens per millisecond, priced with the same row-cost model as § 05, are at least what the native window offers at its current acceptance. Verification is unchanged: greedy candidates must match the target, and sampled ones go through the same rejection test with a point-mass draft q = δ(candidate) (accept with probability p(candidate), otherwise draw from p with the candidate removed). Takeovers are lossless in distribution, but greedy text can differ from MTP-only output.

07 The AR floor

Some requests never benefit: low-acceptance prose on a card where rows are expensive. The same-request AR floor measures instead of guessing:

A tie or an invalid timing also retires. What the floor bounds is the speculative probing (one calibration append and at most three probe windows before the first measured decision), not the full request time. Retired requests are reported as mtp_retired / ar_windows.

08 What it buys you

The gap between the two cards is the row-cost table from § 05 in action: the same head, the same acceptance logic, and roughly twice the price per extra row on the R9700.

09 Limits and switches

10 Sources

• crates/hipfire-arch-qwen4/src/mtp_spec.rs: sampled verification (SampledVerify), penalty history, depth choice, row-cost fits, takeover pricing, AR floor.
• crates/hipfire-generate/src/qwen.rs: penalties installed on every MTP verify row.
• crates/hipfire-config/src/lib.rs: the 5/3/3 n-gram triple for Flash-Next on gfx1151 / gfx1201.
• CHANGELOG, v0.4.1 and the v0.4.1 release notes: measured decode cells and penalty cost.
• Leviathan, Kalman, Matias, “Fast Inference from Transformers via Speculative Decoding” (2023); Chen et al., “Accelerating Large Language Model Decoding with Speculative Sampling” (2023).