hipfire
/learn · prompt reuse

Session cache on a hybrid model

Agents resend almost the same prompt every turn: the same system prompt, the same tool list, a conversation that only grew at the end. Prefilling all of it again is the slowest thing an engine can do on a long context. 0.4.1 gives Flash-Next an engine-owned session cache that keeps many prompts warm at once, and live continuation for the next turn of the same conversation.

The cache is built on #825 and #826 by @fivetide, which introduced the engine-owned session cache and the message-end snapshot work.

01 Why prefix caching is harder on a hybrid model

the idea

In a plain transformer, the state after n tokens is just the first n rows of the KV cache. To reuse a shared prefix you keep those rows and stop there; any cut point works.

Flash-Next is hybrid. Three of every four layers are Gated DeltaNet layers, which keep a fixed-size recurrent state that summarizes the whole prefix; only the remaining layers (QSA attention) keep per-token rows. The recurrent state after token n cannot be recovered from the state after token m > n. So you can only resume where you saved the state, and you have to decide where to save it before you know which prefix the next request will share.

fig · what a resumable point has to contain
  fixed-size state (whole, every time)     append-only rows (grow with the prefix)
  ───────────────────────────────────     ───────────────────────────────────────
  Gated DeltaNet recurrent + conv state    QSA K/V rows
  per-layer n-gram embedding (PLE) state   raw and pooled index keys
  hyper-connection state                   the MTP head's own rows (with MTP)
  current QSA selections
  MTP head state + draft policy (with MTP)

02 Snapshots at chunk boundaries: copy plus delta

how hipfire does it

Prefill already runs in chunks (8,192 rows on Strix Halo, 4,096 on the R9700). Every time a prefill crosses a chunk multiple, the engine takes a snapshot there. A snapshot holds a copy of the fixed-size state, and only the delta of the append-only rows: the rows above its parent, which is the deepest snapshot of the same prefix.

fig · two sessions sharing a system prompt
  session A:  [ chunk 0 ][ chunk 1 ][ chunk 2 ][ tail ]
  session B:  [ chunk 0 ][ different text …            ]

  cache:      S0  = state@c0 + rows[0, c0)          ← shared root
               ├─ S1  = state@c1 + rows[c0, c1)
               │   └─ S2 = state@c2 + rows[c1, c2)  ← A restores S0 → S1 → S2, prefills its tail
               └─ (B's own snapshots at its boundaries)

  restore = walk the chain from the root; every link adds one chunk of rows

Because every link carries one chunk of rows, a snapshot costs about the same at any depth, and sessions that share a prefix (sub-agents with one system prompt, say) share its links. On Strix Halo each Flash-Next snapshot is 513 MiB at any depth; full copies would grow by 481 MiB per chunk, so a 64K-token session needs about 4 GiB for all eight boundaries instead of about 17 GiB.

  • Commit-gated. A snapshot is published only after the client commits the turn, and never depends on live state, so a reset does not drop it.
  • LRU over leaves. A snapshot with children is pinned; eviction removes only leaves.
  • Budget. Default 8 GiB (memory.session_cache_bytes); copies live in system RAM on unified memory and in VRAM on discrete cards, behind a free-memory guard.
  • Engine-owned. hipfire_runtime::session_cache owns keying, planning, eviction, the memory guard, placement and every copy; the model only describes its state parts and prefill boundaries.

03 Cold-exact and session-exact

the idea

There are two different promises a cache can make, and 0.4.1 keeps them apart:

cross-session snapshotlive continuation
whenany request whose prompt shares a cached prefixthe next turn of the same conversation, if its tokens strictly extend what the previous turn consumed
what runsrestore the deepest snapshot, prefill the restprefill only the new suffix, in place, no copy
exactnesscold-exact: equals a cold prefill of the same promptsession-exact: the state descends from decode, as in one continuous conversation
capturesyes, at chunk boundariesno; the shared cache only ever holds cold-exact state

When the live state and the longest cached snapshot reach equally far, the live state wins, and a hit always prefills a non-empty suffix. The hardware test restores a three-link chain and a shared root on both the AR and native-MTP routes and checks every state part, the final logits row and the MTP seed byte-for-byte against a cold prefill. AR and MTP snapshots never cross routes.

04 What it buys you

0.4.1 serve gates · Radeon AI PRO R9700 · Flash-Next
68–78%
of prompt tokens served from cache
plain chat conversations and cross-session prefixes, MTP on and off
0errors
request errors across all 24 serve-gate runs
a 64K-token prompt at max_seq 262,144 on the VMM backend decodes coherently
The cache replaces the earlier single prefix checkpoint, which kept one prompt only: the next request with a different prefix (another session, a sub-agent) overwrote it.

05 Limits and switches

known issues in 0.4.1
  • Tool-call conversations do not reuse the cache. A turn whose history contains earlier tool calls is re-rendered differently from the committed record, so it is not a strict extension and gets no live continuation; TC-shaped cross-session traffic got no hits either (0 of 13 turns on the R9700). Task-eval numbers in the release therefore carry no cache speedup.
  • One request executes at a time per loaded model; concurrent HTTP requests queue.
  • Message-end snapshots (restore at the deepest <|im_end|> instead of a chunk boundary, fivetide's idea) exist but are off by default (HIPFIRE_QWEN4_TURN_SNAPSHOTS=1) until gated.

HIPFIRE_SESSION_CACHE_BYTES=0 turns the cache and live continuation off together; every turn then prefills cold. The previous prefix and radix caches for Flash-Next are removed.

06 Sources