Quantization

How hipfire stores weights and KV cache (design / math). For the operator workflow (hipfire quantize / hipfire-quantize), see QUANTIZE.md. Format-specific specs: quant-formats/mq4-v2.md, quant-formats/mq-v2-family.md, quant-formats/ladder.md, quant-formats/qt-register.txt, quant-formats/hfp4.md.

Truth labels follow INDEX.md. Mutable inventories (CLI flags, registry KV recommendations, arch admissions) are owned by their source pages — this file describes the math and wire contracts.

Weight families at a glance

Production defaults for day-to-day use:

Under standard MQ/HF/HFP/MFP recipes, embeddings are forced to Q8F16 (Q4-grade embedding error compounds). 1D norms / scales stay F16. Direct recipes such as q4k-all intentionally bypass that embedding rule.

Magnum V2 family (qt 44 / 47–50) and MQ4C (qt 45)

Magnum V2 uses a neutral-size 8 B dual half fp16 scale + fp16 zero header per 128 weights, with payload packing unchanged from the matching v1 bit width. Group strides: bits 2/3/4/5/6 → 72/104/136/168/200 B per 256 elements. MQ4C is separate: one per-256 fp16 scale+zero pair at [0..4), zero padding at [4..8), and the 4-bit payload at +8.

Wire implementation ≠ product admission. MQ2V2 loads, decodes, and passes parity, but measured Qwen3.8 ladder KLD is catastrophic (~12–14 nats WT2/v6sel). Do not call MQ2V2 a shipping product body. MQ3–MQ6 V2 cells are measured candidates on the product ladder, not automatic registry cards. Full contracts:

Qwen3.x MoE runtime support covers routed MQ4G256V2 and MQ6G256V2 experts for indexed decode, batched decode, grouped prefill, shared experts, slots, and expert-parallel execution. Uniform and frozen mixed V1/V2 expert tables preserve the exact per-projection dtype tag. This runtime support does not relax the quantizer’s direct-MoE output gate for non-MQ4 V2 formats.

mq4-v2.md, mq-v2-family.md, ladder.md.

Product names (mq4, mq4-xt, mq4 pro, …) are bpw-class + role-lift labels owned by the ladder, separate from codec generation (V2, C) and from wire ids. A product row must satisfy measured model bpw ∈ [N, N+1).

Bonsai (ternary / 1-bit)

Sub-4-bit and research formats (v1 / Lloyd / FP4)

Historical quality note (not a live admission): uniform MQ3/MQ2 local battery on gfx1100 (7900 XTX), Qwen 3.5 / 3.6, master ref c448d5e (2026-04-30). MQ3 was fluent on-topic at 9B and 27B across a 4-prompt coherence set; 4B was partial collapse; 0.8B gibberish. Uniform MQ2 collapsed at every size tested. Treat as scoped historical provenance, not a product floor. MQ3-Lloyd / MQ2-Lloyd remain research-gated until product validation. Separately, the 2026-08-20 dense Qwen3.8 MQ V2 ladder rejects MQ2V2 on KLD while keeping wire support.

Prefill / WMMA runtime batch eligibility (weights)

is_batchable_la(dtype, arch) is runtime batch eligibility only — it is not product admission and is not an empty/full admission registry. llama and qwen35 each own their own predicate and are not currently in lockstep (crates/hipfire-runtime/src/llama.rs, crates/hipfire-arch-qwen35/src/qwen35.rs).

Always batch-eligible (both owners): MQ4G256, HFQ4G256, MQ6G256, HFQ6G256, Q8_0. (qwen35 also treats ParoQ4G128 / F32 as always-ok for the PARO probe path.)

Uniform MQ3 ships WMMA residual / fused GEMM kernels on RDNA3+ and is batch-eligible on the arches above. Do not claim “no MQ3 WMMA.” Lloyd batch eligibility is owner-specific (qwen35 yes on listed arches; llama no).

FWHT rotation (the M in MQ)

Each 256-wide group is transformed by two fixed ±1 sign diagonals and a Walsh–Hadamard butterfly, then scaled by 1/√256 = 1/16. Encoder and runtime share the same contract (cpu_fwht_256 / fwht_forward_256 / rotate_x_mq):

R = D2 · H · D1 / 16
  signs1 → Hadamard butterfly → ×(1/16) → signs2

quant:       w′ = R w   →  quantize w′
inference:   x′ = R x   (same R, not an inverse)
             y  = (w′)ᵀ x′  = wᵀ x     in exact arithmetic

Sign tables use fixed PRNG seeds 42 (D1 / signs1) and 1042 (D2 / signs2), shared by quantizer and runtime. Cancellation holds in exact arithmetic; fp32 mixed-precision paths accumulate rounding drift (typically below per-block quantization error, but not bit-exact identity).

Why it matters: outliers spread across the group, narrowing dynamic range before 4-bit (or lower) quantization. The rotation was calibrated against Qwen 3.5 / 3.6 (DeltaNet hybrid) weight distributions — MQ4 targeting Q8-class quality at Q4 bandwidth on that family is the central product claim.

On Llama-style dense models the math still cancels, but there is typically no quality lift over plain HFQ4 — only the extra rotate_x_mq cost. Practical rule: MQ for Qwen 3.5+; HFQ for everything else. CLI defaults match (safetensors → mq4, GGUF → hf4).

Header layout (HFQ/MQ G256 affine blocks)

v1 (qt 6/13/15/17/18/…)

Per 256 elements, 8-byte header + packed payload. HFQ4 and MQ4 v1 both store an f32 affine scale and an f32 minimum (not FP16-in-f32):

• scale: f32
• zero / min: f32
• data: bitwidth × 256 / 8 bytes (128 for 4-bit, 96 for 3-bit, 192 for 6-bit, …)

V2 (qt 44 / 47–50) and MQ4C (qt 45)

Same 8-byte header budget and payload offset +8, different header encoding:

• MQ*G256V2: fp16 scale+zero for half 0 (weights 0–127) at [0..4), half 1 at [4..8); payload packing identical to the matching v1 bit width.
• MQ4CG256: one packed fp16 scale+zero at [0..4), mandatory zero pad [4..8), 128 B nibbles at [8..136) (v1-compatible geometry).

Lloyd variants replace the uniform grid with a per-block (or per-tensor) fp16 codebook plus packed indices; byte sizes differ (e.g. MQ3-Lloyd 112 B/group, MQ4-Lloyd 160 B/group).

KV cache

K and V are quantized differently: K enters softmax (errors exponentiate); V is already attention-weighted. Live modes store V as Q8 and compress K:

Default resolution (crates/hipfire-cli/src/main.rs):

1. HIPFIRE_KV_MODE env
2. Per-model config kv_cache
3. Registry default_kv_mode when config is auto
4. Else q8

There is no live per-arch archDefaults table (removed). Compressed modes are opt-in via registry cards or explicit config — not guessed from GPU VRAM. Override: hipfire config set kv_cache <mode> or HIPFIRE_KV_MODE=<mode>.

Optional adaptive downshift (kv_adaptive: off | conservative | balanced | aggressive | advanced:k=…,v=…) can lower V and K tiers as context grows; see CONFIG.md. Works best when K starts at a higher fwht tier.

HFQM container (.hfq / .mq4 / .hf4 / …)

Extensions are naming convenience; the on-disk magic is always HFQM:

0x00  "HFQM"              (4 bytes)
0x04  version             (u32 LE = 1)
0x08  arch_id             (u32 LE — see architecture-ids.md)
0x0C  n_tensors           (u32 LE)
0x10  metadata_offset     (u64 LE)
0x18  data_offset         (u64 LE — 4096-aligned)

[metadata_offset .. data_offset):
  metadata_json (UTF-8: config, tokenizer, gguf_meta, source, …)
  tensor_index:
    n_tensors: u32 LE
    for each tensor:
      name_len: u16 LE
      name: utf-8
      quant_type: u8      (see QuantType table below)
      n_dims: u8
      shape: u32 LE × n_dims
      group_size: u32 LE
      data_size: u64 LE

[data_offset .. EOF]: tensor bytes in index order

HfqFile::open mmaps the file; tensor slices feed HIP upload without an extra host copy. Bundled .mq4-mtp files append an MTP HFQM section + HFBNDMTP trailer; trunk loaders ignore the trailer and see only the trunk.

QuantType IDs (encoder / loader contract)

Source of truth: quant-formats/qt-register.txt.

Current reserved wire IDs: 23 and 25–27 only. IDs 22, 29, and 30 were reassigned (TidI32, PARO4G128T, MQ4G256Lloyd). qt 42–43 and 46 are not live product rows in the register (43 burned by retired SEL; 46 is hierarchical mq4_k design-only). Do not invent slots — edit qt-register.txt to claim a number.

PARO4G128 probe payload

quant_type=28 is not row-major HFQ4/MQ4. Each tensor stores ParoQuant buffers in order:

qweight        int32 [K, M/8]
qzeros         int32 [K/128, M/8]
scales         f16   [K/128, M]
pairs          int16 [8, K]
theta          f16   [8, K/2]
channel_scales f16   [K]

Runtime contract:

x_rot = rotate(x, pairs, theta, channel_scales)
y     = awq_w4_gemv(x_rot, qweight, qzeros, scales)

Verify imports with python3 scripts/astrea.py paro-oracle --source PARO_SAFE_DIR --hfq MODEL.hfq --module MODULE --pretty before quality or perf claims.

Why a custom format

llama.cpp GGUF Q4_K_M is similar on disk size and in-place GEMV cost. hipfire’s gains are engineering, not a new algorithm class:

1. Fused projections — QKV / QKVZA / gate+up fused launches for HFQ4/MQ4 (and siblings) cut launch overhead on decode.
2. WMMA prefill — batched GEMM on RDNA3+ WMMA for MQ/HFQ (and MQ3/HFP on eligible arches) amortizes dequant across prompt tokens.

Published speedups belong in measured owners (BENCHMARKS.md, perf-checkpoints); do not promote undated multiples here.

Related

• Operator quantize: QUANTIZE.md
• Magnum V2 / family / ladder / register: quant-formats/mq4-v2.md, quant-formats/mq-v2-family.md, quant-formats/ladder.md, quant-formats/qt-register.txt
• HFP4 wire spec: quant-formats/hfp4.md
• Config / env KV knobs: CONFIG.md, env-vars.md
• Models / registry: MODELS.md
• Arch IDs: architecture-ids.md
• Sources: crates/hipfire-quantize/src/ (QuantType, encoders, pipeline), crates/hipfire-runtime/src/{hfq,llama,weight_backend}.rs, crates/hipfire-cli/src/main.rs, crates/hipfire-config/src/lib.rs