hipfire
/learn · tokenizer

Tokenizer fidelity: the whitespace lookahead

A model only behaves the way it was trained if it sees the same token IDs it was trained on. 0.4.1 fixes a long-standing difference between hipfire's byte-level BPE tokenizer and the Hugging Face reference: how a run of whitespace before a word is split. It is a CPU-side correction, so it applies on every GPU and every quantization.

01 Pre-tokenization in one paragraph

the idea

Byte-level BPE tokenizers (GPT-2 style, used by the Qwen family) first cut text into pieces with a regular expression, then run BPE merges inside each piece. Merges never cross a piece boundary, so where the regex cuts decides which tokens are even possible. The pattern is an ordered list of alternatives; at each position the first alternative that matches wins. 

(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

The interesting alternative is the second-to-last one, \s+(?!\S): “a run of whitespace that is not followed by a non-space character.” Before a word, the regex engine backs off by one character to satisfy the lookahead, which leaves the last space of the run for the word that follows.

02 Worked example

Four spaces of indentation in Python, with and without the alternative:

fig · pre-tokenizer pieces for "def f():\n return 1"
  with \s+(?!\S)  (Hugging Face, 0.4.1):
     'def' ' f' '():\n' '   ' ' return' ' ' '1'
                         └ 3 ┘  └ space joins the word

  without it (before 0.4.1):
     'def' ' f' '():\n' '    ' 'return' ' ' '1'
                         └ 4 ┘  └ bare word

Same text, different pieces, so different token IDs: the model sees " return" (the token it was trained on in this position) instead of "return" after a different whitespace token. Code, tool-call JSON and aligned text are full of whitespace runs, so agent prompts hit this constantly.

Pieces shown are the output of the pattern above, and of the same pattern with the \s+(?!\S) alternative removed, on this string.

03 What changed in 0.4.1

how hipfire does it
  • The lookahead alternative is honored. A whitespace run before a non-space leaves its last Unicode character for the following alternative; newline alternatives and trailing whitespace keep their own boundaries.
  • The model's own split pattern is used. The runtime reads the Hugging Face Split regex embedded in the model rather than imposing one pattern (with three-digit number grouping) on every byte-BPE model.
  • The declared NFC normalizer is honored for byte-BPE inputs, and the pre-tokenizer and normalizer now take part in the persistent cache identity.

The previous code applied one pattern to every GPT-2-style tokenizer: Qwen2, Qwen3, Qwen3.5 / 3.6 / 3.8 dense and MoE (including the 27B), Flash-Next, and dots.ocr. SentencePiece tokenizers are unchanged.

04 How it is tested

0.4.1 commits an offline Hugging Face corpus with exact reference IDs and a test, hf_tokenizer_corpus, that checks every ID, not just token counts:

corpus partcases
reconstructed Hermes agent prompts138
TC requests + Strix Halo TC captures55 + 14
benchmarks/prompts/*.txt44
synthetic whitespace / Unicode cases694
total (868,076 reference tokens)945

scripts/gen_hf_tokenizer_corpus.py regenerates the corpus from local captures with the pinned Hugging Face tokenizer and chat template, and records input MD5s alongside the IDs.

after 0.4.1

A broader differential test, which compares every shipped byte-BPE family against Hugging Face with seeded fuzz input, was written after the 0.4.1 release and is not part of it. 0.4.1's guarantee is the fixed corpus above.

05 What it buys you

0.4.1 · reconstructed Strix Halo Hermes prompts
120/ 138
prompt token streams change with the fix
96 change length; 24 keep the count but differ
42/ 138
prompts where the old token counts matched Hugging Face
the corrected tokenizer matches the reference IDs exactly
945
corpus cases checked ID by ID
868,076 reference tokens
These are prompt-input changes: the model now receives the IDs its reference tokenizer produces. 0.4.1 makes no claim that this by itself changes generation quality, tool-loop outcomes or speed.

06 Limits

what to expect when upgrading
  • There is no switch: prompt token IDs change for any input with whitespace runs before non-space text.
  • Generation for the same text is not expected to match older releases.
  • Fixed-ID references (KLD references, kernel identity pins, frozen pre-tokenized fixtures) are intentionally unchanged, because they do not re-encode text.

07 Sources