CLI

Audience: operators driving the native Rust hipfire binary (crates/hipfire-cli). Run hipfire <cmd> --help (or hipfire help <cmd>) for the live flag text. This page indexes subcommands and file locations; it does not duplicate the full model registry or every config key.

Bare hipfire (no subcommand) is equivalent to hipfire tui: it launches the terminal UI and requires the hipfire-tui binary (HIPFIRE_TUI_BIN, ~/.hipfire/bin/hipfire-tui, the workspace target/release, or PATH). When that binary is missing the command exits with an error naming the build command; it does not print a command list (use hipfire help). Config keys: CONFIG.md. Env vars: env-vars.md. Models: MODELS.md.

Model lifecycle

Tags resolve through the dynamic registry + aliases. Authoritative live list: hipfire list -r. The native binary embeds registry/v1.json; the curated editing surface is registry/models.json. At startup it may refresh the v1 payload into ~/.hipfire/registry.cache.json (24h). Offline or invalid remote payloads fall back to cache then the embedded registry. Pin the bundle with HIPFIRE_NO_REGISTRY_FETCH=1.

Flash-Next MQ4 XTS tags (short name qwen3.8:flash, alias for qwen3.8:flash-next) automatically migrate the exact SHA-verified installed qwen3.8-flash-next-gptq3.mq4 to qwen3.8-flash-next.mq4-xts by a no-clobber hardlink, preserving the old name. External-storage symlinks are preserved; when linking is unsupported or read-only, the verified old path is used. This does not download or copy the payload. Conflicting canonical files fail clearly; the historical RTN .mq4 artifact is never migrated. See MODELS.md for the compatibility window.

Retired Flash-Next tags

qwen3.8:flash-next-rtn-asym (qwen3.8-flash-next.mq4) and qwen3.8:flash-next-mq6q8-pleq8 (qwen3.8-flash-next.mq6q8-pleq8) are retired. hipfire pull refuses either tag or bare filename before auth/network, including an already installed copy and --force, with this one-line error: <tag> is retired; the file still runs if you have it; replacement qwen3.8:flash-next.

hipfire run and hipfire serve accept the old tag, bare filename or installed model path after SHA-256 and size verification, without redownloading it. A copy elsewhere loaded by explicit path remains a plain-file load, not a name-based retired identity. A missing retired file errors instead of pulling (serve: HTTP 404). hipfire list marks installed models [retired] (JSON: "retired": true); the available catalog (including list --remote) omits them.

Compiled RETIRED_MODELS pins preserve the original repo/file/SHA-256/size; neither artifact had sidecar, overlay or hotset pins. Verification reuses <HIPFIRE_HOME>/cache/verified-artifacts.json (VerifiedCache). This does not create new overlay/hotset identities or delete or alter any Hugging Face file. Both entries are removed from the raw remote registry and new bundled/loaded available views, without adding retirement wire fields. Code inspection shows that 0.4.2 rejects the generic ple schema and falls back to cache or bundle; this is not an exercised old-binary test. Monolithic Hugging Face files must stay published while 0.4.2 clients exist: their bundles still reference them. The obsolete local-only qwen3.8:flash-next-mq4-xts-zab entry is removed outright from the registry with no retired pin; its trunk tier ships as the default overlay on the stock tags. The split flash-next / -mq4 / -gptq3 / -mq4-xts aliases, MQ2/MQ3 tiers and overlay pins are unchanged.

Flash-Next split PLE layout

A split registry entry (file = <name>-base.<ext>, e.g. qwen3.8-flash-next-base.mq4-xts) ships the model without its 128 PLE n-gram shards plus swappable row files next to it: <base>.ple-q8 (exact Q8F16 rows, ~54.4 GB) and <base>.ple-q4 (4-bit Lloyd-Max rows, ~28.8 GB, bit-identical in decode to the q4 hipfire ple derive makes locally from the q8 rows).

• Pull. hipfire pull <tag> [--ple auto|q4|q8|all|none] downloads the base plus the chosen sidecar(s) after printing the plan (files, sizes). auto (default) follows memory.ple_format (HIPFIRE_PLE_FORMAT), whose own auto is q4 on gfx1151 and q8 elsewhere; an unknown GPU arch (or no GPU) resolves to q8. On a host with several GPU archs auto fetches the union of every detected arch’s default (a gfx1100 + gfx1151 host plans q4 and q8). Unknown registry formats (a future q2) are ignored with a note; an unpublished lossy format falls back to the published q8, an explicit --ple q8 never falls back. none fetches the base only. All files are SHA-256 verified; a failed sidecar fails the pull. Without --ple, a pull on a terminal asks which rows to fetch (preselecting what auto fetches, both on a mixed host, or q4 when host RAM cannot hold the base plus the q8 rows); --yes or a non-terminal stdin takes auto.
• On-demand. run, bench and serve download a missing sidecar before loading (ple: downloading <file> (<GB>) for memory.ple_format=<..>): q8 when q8 is requested; q4 only when neither <base>.ple-q4 nor <base>.ple-q8 is on disk. With only the q8 rows present, a q4 load is refused until you run hipfire ple derive --format q4 <base>: a load never derives. auto uses whichever of the two is present and downloads the arch’s preferred format only when neither is. A serve request naming such a model downloads only when serve.allow_request_pull is on; otherwise the load is refused with the pull command.
• No forced migration (grandfathered monolithic). A user who already has the monolithic qwen3.8-flash-next.mq4-xts (or its legacy …-gptq3.mq4 name) keeps serving it under the split tags: a compiled-in pin (the split base’s identity, sha256 eafaff94…) lets a SHA-verified local monolithic serve the tag while the split base is absent, so hipfire pull <tag> downloads no base (Already satisfied (monolithic): <path> (serves <tag> without the split base), plan line present (already satisfied by the monolithic; no base download)). The exact q8 rows are embedded in it, so q8 is never fetched for it; a planned q4 is linked beside it, and the monolith-bound overlay and hot set are fetched when missing. One note says hipfire ple split <tag> or hipfire pull <tag> --force gets the split layout. Verification is cached (<HIPFIRE_HOME>/cache/verified-artifacts.json: path, device, inode, size, mtime, sha256): an unchanged file is not hashed again on later loads.
• hipfire ple derive [--format q4] [--rank PATH] <model> writes <model>.ple-q4 from the model’s Q8 rows (<model>.ple-q8 when present, else the embedded rows), ordered by --rank (default <model>.plehot when present, else row order). --format q8 is an error: q8 rows are the model’s own rows or the file hipfire ple split writes. Progress (percent, GB, MB/s) prints at most every ~2 s; the summary lists output, sizes, elapsed time and the payload_sha256 / q8_rows_sha256 digests.
• hipfire ple split [--out PATH] [--keep-original] <model> converts a monolithic model into <base> (default: -base inserted before the extension) + <base>.ple-q8, verifies both outputs, then removes the original (and its hardlinked legacy spelling) unless --keep-original. It refuses to overwrite existing outputs and a non-monolithic input. <model>.plehot* files belong to the monolithic layout and are left in place with a note. The summary prints base_index_sha256 and q8_rows_sha256.
• hipfire ple rebind --old-base PATH --new-base PATH --source PATH is for a repacked monolithic base whose non-PLE weights changed but whose embedded PLE rows did not. It validates the source q4 sidecar against the old base, compares both bases’ PLE descriptors and Q8 payloads, verifies every q4 segment digest, and copies the unchanged q4 payload with only the new base-size/index binding. After readback and attachment validation it atomically replaces <new-base>.ple-q4 with a separate inode, including when that destination was a hardlink to the source. The source file is never rewritten. Changed PLE bytes/layouts or corrupt digests are refused; split bases are not supported. A stock sidecar must not simply be hardlinked to a smaller requantized base: its binding will correctly fail the loader.
• rm and listings. rm also removes <model>.ple-q4, <model>.ple-q8, their .plehotrows packs and <model>.plehot. *.ple-q4, *.ple-q8, *.plehot and *.plehotrows files are sidecars: never listed or served as models.

Flash-Next trunk overlay

The four stock Flash-Next MQ4 XTS tags (qwen3.8:flash-next, -mq4, -gptq3, -mq4-xts) share one base artifact (sha256 8b15b6fe…, 125288544792 B). The client carries a compiled-in pin for that base, naming the trunk overlay qwen3.8-flash-next.mq4-xts.trunk-overlay (the canonical base file name plus .trunk-overlay; 305543168 B, sha256 df49bc0a…) in the hipfire-models/qwen3.8-flash-next repo. There is no registry schema change, so older clients keep parsing the published registry. The overlay holds the 108 GDN in_proj_{z,a,b} projections as pre-encoded symmetric QT44, bound to the base by size and index SHA-256. The Qwen4 loader substitutes them for the base’s MQ6 tensors on gfx1151 by default; HIPFIRE_QWEN4_TRUNK_OVERLAY=0 restores the base weights and =1 applies the overlay on any architecture.

• Pull. hipfire pull <tag> downloads and SHA-256 verifies the pinned overlay with the other sidecars; a valid file already on disk is not downloaded again, a stale one is refreshed. The loader applies the overlay only on gfx1151, so the pull fetches it only when a detected GPU is a gfx1151 (a mixed host with one counts) or none is detected (the pull cannot know where the model will run); on a host whose GPUs are all other architectures it prints Skipping Trunk overlay sidecar and downloads nothing for it, and the interactive summary does not list it. HIPFIRE_QWEN4_TRUNK_OVERLAY=1 (apply on any architecture) keeps the download everywhere. After adding a gfx1151 card, run hipfire pull <tag> again to fetch it. As for other sidecars, a failed overlay download prints a warning and the pull still succeeds (the base loads unchanged). Any other registry entry (e.g. the RTN-asym file) has no overlay.
• rm and listings. rm <tag> removes the overlay with the base (<file>.trunk-overlay). *.trunk-overlay files are sidecars: never listed or served as models.

Flash-Next expert hotset

The same four stock Flash-Next MQ4 XTS tags (one base artifact, sha256 8b15b6fe…, 125288544792 B) also have an expert-hotset sidecar. As for the trunk overlay, the client carries the pin, not the registry: it names qwen3.8-flash-next.mq4-xts.expert-hotset (the canonical base file name plus .expert-hotset; 217676 B, ~213 KiB, sha256 7f6760e1…) in the hipfire-models/qwen3.8-flash-next repo, matched by the base’s exact size and SHA-256. registry/v1.json is unchanged, so older clients keep parsing the published registry.

• Pull. hipfire pull <tag> selects the hotset by default. On a terminal the interactive pull prompt has an independent default-on yes/no choice, Expert hot set (R9700 decode, 213 KB), shown in the summary only when selected (--yes and the TUI’s Models-tab pull take the default); the “213 KB” is the rounded display label, the pinned file is 217676 B (~213 KiB). The hotset is downloaded and SHA-256 verified with the other sidecars to the sidecar path of the local base file, which may be a renamed local file as long as it matches the pinned base identity; a valid file already on disk is not downloaded again, a stale one is refreshed. As for other sidecars, a failed hotset download prints a warning and the pull still succeeds. Any other registry entry (e.g. the RTN-asym file) has no hotset.
• Runtime. The hotset applies by default on a single R9700 (gfx1201) when the sidecar is present and bound to the base by size and index SHA-256.
• rm and listings. rm <tag> removes the hotset with the base (<file>.expert-hotset). *.expert-hotset files are sidecars: never listed or served as models.

Inference

hipfire run flags

Flags may appear before or after the model. CLI help and the native typed schema list stored globals; effective run sampling is not “global default unless overridden.” The Rust request resolver transmits only explicit CLI flags, per-model overlay values, or registry recommended_settings. Otherwise the field is omitted so daemon/HFQ/arch fallback applies. Global-only temperature / top_p / repeat_penalty are not sent. Chat is the exception (global session snapshot — CHAT.md).

Resolution ladder for speculation: CLI flag > env > per-model > global > registry card > built-in default. On run, --spec replaces the selector the config and environment layers resolved (hipfire config explain speculation.mode prints the winning source). dflash_mode itself defaults to off in the schema — see CONFIG.md — except that Qwen3.8-27B tiers with a registry-declared draft resolve auto through the registry layer (CONFIG.md); an explicit dflash_mode=off (per-model, global or env) wins. Pulling a draft for any other model does not enable DFlash.

--stats illustrative output (values are placeholders, not a benchmark): [stats] tokens=256 decode=123.30 tok/s ttft=42.1 ms tau=7.15. tokens counts committed daemon completion IDs, including reasoning, not printed characters. decode is decode throughput, not wall-inclusive tok_s; ttft is time to first token. tau is included only for speculation when available. Missing, invalid or nonfinite metrics show n/a, never a substituted wall rate or zero. A continuous-batch response may legitimately have decode=n/a.

hipfire run qwen3.5:9b "What's 2+2?"
hipfire run qwen3.5:9b "What's 2+2?" --stats
hipfire run qwen3.5:9b --spec ngram "Repeat verbatim: ..."
hipfire run qwen3.5:27b -md ~/.hipfire/models/qwen35-27b-dflash-mq4.hfq "..."
HIPFIRE_LOCAL=1 hipfire run qwen3.5:4b "..."   # skip HTTP; always local spawn

Local-forcing (skip a healthy serve): HIPFIRE_LOCAL truthy, --image, --kv-mode, --kv-backend, --kv-k, --kv-v, --max-seq, --head, --tp above 1, --spec/--speculation, --model-draft, --vision, --draft-max, or --dspark-conf-threshold (exact list: run_should_force_local in crates/hipfire-cli/src/main.rs). JSON and non-streaming responses are supported by the native HTTP service and do not by themselves force a local daemon.

KV load flags (run / serve / bench)

Shared by hipfire run, hipfire serve, and hipfire bench (including --matrix). Full config surface: CONFIG.md.

Qwen auto / unset mode: native fp8 (both K and V) on exact gfx1201 single GPU when the load is native-eligible — H24/Hkv4/D256 dense or H16/Hkv2/D256 A3B attention, no PP/TP/EP/slots split, no kv_adaptive, no CASK sidecar; q8/q8 everywhere else, including gfx1100 and gfx1151. Native bf16 stays H24 dense-only. Native modes report kv_mode=fp8 or bf16 in loaded/diag/bench JSON; no synthetic V=q8 axis is reported. Non-Qwen family defaults are unchanged (e.g. Maple BF16, DeepSeek compressor F32, Gemma layered policy).

K/V precedence (split modes only): CLI flag > per-model config > global config > registry / arch default. Mode seeds the pair; --kv-k overrides K and --kv-v overrides V. An authored --kv-mode fp8/bf16 refuses every axis override, including --kv-v q8; an auto-selected native pair can be replaced only when both axes are supplied. Full refusal rules: CONFIG.md.

Backend precedence (forced choices): CLI --kv-backend > per-model memory.kv_backend > global memory.kv_backend. Omitted backend stays automatic VMM-preferring policy, not an authored pin.

hipfire serve flags

hipfire img flags

Run in a fresh daemon process; the prompt is the positional text (a built-in sample prompt when omitted). --steps defaults to the model’s own default (4 for step-distilled flux.schnell:1, 28 for guidance-distilled flux.dev:1).

Model resolution accepts an HFQ component pack (trunk file) — the t5/clip/vae sidecar packs are discovered next to it by sibling name (<stem>-t5.hfq etc., or shared t5-xxl.hfq / clip-l.hfq / vae.hfq). hipfire pull flux.schnell:1 fetches the trunk + all three sidecars at once.

hipfire embed flags

Runs in a fresh daemon process. The production target is an embedding checkpoint loaded as an HF safetensors directory (e.g. a pplx-embed-v2-late-0.6b snapshot), or a raw-dtype .hfq built from one with hipfire-quantize (the quantizer keeps the norms, the 128-d Dense head and the vision tower raw, folds 1_Dense/model.safetensors into the trunk, and auto-includes the tower).

EmbeddingGemma-2 (arch 18) snapshots — e.g. a google/embeddinggemma-2 directory, whose quantized form writes a .vl tower sidecar — are served by the same command. They answer with one pooled vector (n_tokens: 1) instead of arch 17’s token matrix, and --image / --audio / --video reach their vision and audio towers (video drives the vision tower frame by frame) when the sidecar is resident.

Token limits are per side (query_length 1024, document_length 4096 on the 0.6B); text is truncated from the end with the marker always kept. The document side drops punctuation token vectors (the ColBERT skiplist); the query side keeps them. Scores are late interaction (Σ max dot), so they are not cosine similarities and are only meaningfully comparable between the same query across documents.

hipfire embed ~/.hipfire/models/pplx-embed-v2-late-0.6b \
  --query "what is the capital of France" --doc "Paris is the capital of France"
hipfire embed <model> "encode this document" --kind document --json
hipfire embed <model> --image chart.png "the caption" --json

The same surface is exposed over HTTP as POST /v1/embeddings (OpenAI-shaped; the embedding array is the flat n_tokens × dim matrix and each data object carries n_tokens, with an optional "kind": "query" | "document"). input takes a string, a one-element list, the object form ({text, image_base64}, or {text, images_base64: […]} for several — the same spelling the JSONL embed verb takes), or the typed content array ([{"content": [{"type": "text", "text": …}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,…"}}]}]) — one content object per request, text parts concatenated in order, image parts decoded at the route. A bare content-part array without the {"content": …} wrapper is refused with the wrapper it should have used. Interleave media by putting one <|image|> marker per image in the text (arch 18); a text with no marker gets its image runs appended before EOS, and a marker count that disagrees with the image count is refused. Audio takes the OpenAI input_audio part ({"type": "input_audio", "input_audio": {"data": "data:audio/wav;base64,…", "format": "wav"}}, WAV/RIFF only, format optional) and the JSONL audios_base64 list, with one <|audio|> marker per clip under the same count rule; a text with no marker gets its audio runs appended after the image runs. Audio needs the checkpoint’s audio tower (the same .vl sidecar as vision): a vision-only load refuses a clip by name rather than encoding the text alone.

Video takes the llama.cpp-shaped input_video part ({"type": "input_video", "input_video": {"data": ["data:image/png;base64,…", …]}}, one entry per frame in display order, format optional and PNG/JPEG only) or the JSONL videos_base64 field — one frame list per video, e.g. {"text": "… <|video|> …", "videos_base64": [[frame0, frame1, …]]}. One <|video|> marker per video under the same count rule, and the daemon samples the frames itself (the video processor’s max_frames 32 budget, uniform stride) before running the vision tower once per frame; a frame that is not a PNG/JPEG image is refused with the ffmpeg extraction to run.

Configuration

Do not inventory every key here — CONFIG.md owns defaults and ranges. Notable defaults from source: dflash_mode=off (schema default; Qwen3.8-27B tiers with a paired draft resolve auto through the registry layer), speculation=auto, thinking=on, reasoning_effort=auto (no preference: an explicit non-auto config/CLI/HTTP value wins; when every layer is unset/auto, Qwen3.8 and Ornith 1.5 use hipfire’s medium default, so reasoning_effort = "xhigh" restores the official-template default), schema default thinking_budget=med (legacy named-cap route only; dropped+warned on effort-native models), max_tokens=4096, idle_timeout=300. Qwen Jinja contracts additionally accept an explicit integer think cap; other families do not synthesize a force-close mechanism. Reasoning axes and family examples: CONFIG.md, SERVE.md.

Quantization and calibration

quantize (summary)

Supported CLI formats include mq4, mq6, q8/q8f16, hf4/hf6 and hfq aliases. Graded MoE recipes need the quantizer binary directly — QUANTIZE.md.

sidecar-gen (summary; deprecated since 0.4.0, removal in 0.5.0)

Benchmarks and diagnostics

Perf claim protocol (warmup, fresh-process, noise): methodology/perf-benchmarking.md. Published tables are measured/historical: BENCHMARKS.md.

hipfire bench flags (summary; hipfire bench --help is authoritative):

Bench concurrency (--concurrency)

hipfire bench <model> --concurrency 1,2,4,8 [--backend batch|noslots|slots|both] [--workload stateless|multiturn|both] [--bench-shape <shape>] [--num-prompts N] [--temperature T] [--reasoning-on] [--runs N] measures serving throughput with a closed-loop sweep at each concurrency level K (--bench-shape defaults to steady; hipfire bench --help lists the other shapes and the flags they need):

• Closed loop at K. K client lanes drain one FIFO of N requests. A lane sends its next request the moment its previous one ends, so there is no burst barrier and no lane idles until the queue is empty. Request i is prompt i mod len of the shape’s list, so every K sends the same multiset in the same order.
• Warmup discarded. Before every sample, 8 separate warmup prompts run closed-loop at the same K with 64-token outputs. No measured prompt is ever a warmup prompt.
• No prefix-cache hits. Each request’s first message starts with a Q<run>-<i>. tag, unique per --runs round and request. Every K of a round therefore sends the same tokens, and a later round’s repeat never hits the daemon’s prompt cache. cached reports any prompt tokens the daemon served from cache.
• Chat turns, neutral sampling. Requests carry messages (plus prompt.system, as hipfire serve injects it), thinking off unless --reasoning-on, and neutral penalties (repeat_penalty 1.0, presence_penalty 0, min_p 0).
• Levels are interleaved across --runs (1,2,4,8,1,2,4,8,…), and the table reports each metric’s median over the runs.

The multiturn workload makes each request a two-turn chat. The follow-up (“Explain that again in one sentence.”, with the first answer as the assistant turn) is sent on the same lane as soon as the first turn completes, and both turns count. On slots, a sample whose follow-ups recorded no prefix-cache hit is reported [invalid] and dropped.

--temperature overrides the shape’s temperature (default 0 for steady). The daemon has no ignore_eos, so steady fixes the output length by prompt choice: its prompts ask for answers longer than 256 tokens. Check that the finish column reads length only. Sampled requests (temperature > 0) are not batched: the VMM route serves greedy requests only, so a sampled sweep measures the singleton route queued K deep.

Each sample prints one line on stderr: output tok/s, steady tok/s, TTFT and TPOT p50/p99, in-flight, τ, ok/total, tokens and duration, finish reasons, routes (execution_mode/batched=), cached prompt tokens, mean prompt tokens and the md5 of the decoded texts. The final table has one row per (backend, workload, K):

• out tok/s: output throughput, output tokens ÷ (last completion − first send). The median of the runs, with min–max in brackets.
• steady tok/s: tokens whose arrival falls in the steady window ÷ the window. The window runs from the moment all of the first K requests have a token to the first completion after the last send.
• TTFT: first token − send. TPOT: (completion − first token) ÷ (tokens − 1). Both are the medians of the per-sample p50/p99.
• in-flight: achieved concurrency, Σ request latency ÷ duration.
• τ: output tokens per verify window, token-weighted over the sample.
• tokens: median output tokens per sample.
• finish, batched, cached, errors: summed over all samples. A failed request counts no tokens.

A daemon that sends no frame for 300 s fails the sweep. Its in-flight requests are cancelled first.

Bench context (max_seq)

--max-seq N sets the benchmark load’s KV context, within the memory.max_seq bounds (512–1048576), and replaces any configured memory.max_seq.

Without it, --matrix and --redline size the load to the run instead of using the configured or automatic context. The run needs the largest --pp, or the largest --ctx plus --tg (and --sustained-ctx plus --sustained-tg when sustained rows run), whichever is larger, plus the 32 positions of headroom the daemon’s probes keep. That is raised to the memory.max_seq minimum of 512 and capped at the model’s declared context (max_position_embeddings in its metadata). --pp 8192 --ctx 128 --tg 32 loads max_seq 8224.

• A shape past the model’s declared context fails before the daemon starts, naming the run’s need and the model’s limit. This applies with or without --max-seq.
• A --max-seq below a matrix run’s need fails the same way, naming both numbers.
• Past the declared context, a --max-seq that holds the run is sent as given, and the daemon’s admission is the bound. For example, Qwen4 refuses anything above its admitted 262144.

The standard, --ttft, --exp and --concurrency paths have no fixed shape, because the prompt is tokenized by the daemon. Without --max-seq they keep the configured or automatic context.

Every JSON report has a max_seq object:

• effective: what the daemon loaded.
• requested: the value sent, or null.
• source: flag, run_shape, config (an authored memory.max_seq) or automatic.
• run_need: the matrix need, or null.
• model_limit: the declared context, or null.

The bench also prints a max_seq: line.

Bench matrix topology

--matrix measures the model on whatever topology the load admitted: one card, pipeline-parallel (topology.pp, 2 or more stages), dense tensor-parallel (--tp / topology.tp) or expert-parallel (EP). Admission is serve’s own, so a load serve refuses is refused by the bench with the same message. The rows are synthetic: fixed token ids on the plain prefill route and a primed synthetic decode context. They time the loaded topology’s forward, not a prompt, speculation or the HTTP path, and a multi-card load’s rows carry that load’s cross-card traffic. Time a real prompt with --prompt-file (see the standard bench).

The --matrix JSON gains a topology object, taken from the daemon’s own prefill_result / decode_result answers and not derived from the flags:

A single-card load is {"pp":1,"tp":1,"ep":1,"cards":["0"]}; a PP3 load {"pp":3,"tp":1,"ep":1,"cards":["0","1","2"]}; tp and ep are reported separately, never merged. All rows of one run must report the same topology (including the card list), and topology.pp must equal the report’s pp (the loaded value), otherwise the bench fails instead of labeling the run. A daemon that reports no topology yields "topology": null, never an assumed single card. Every other --matrix field is unchanged.

Bench daemon lifetime

hipfire bench owns the daemon it spawns (and, through it, the machine-wide per-GPU flock of the cards the load reserves). The daemon is killed and reaped when the bench returns or fails, and when the bench receives SIGINT, SIGTERM or SIGHUP; an interrupted bench prints hipfire bench: interrupted; terminated N owned daemon(s) and exits 130. Reaping completes before the CLI exits, so a harness that waits for hipfire bench can start the next arm on the same cards immediately. Only daemons this bench process spawned are touched. SIGKILL of the CLI itself cannot be intercepted and leaves the daemon running, so harnesses should send SIGTERM to the CLI first and escalate to SIGKILL (on the whole process group) only after a grace period.

There is no load timeout in the CLI, the client or the daemon: a cold multi-GPU load (first-use kernel compilation, 100+ GB of weights) runs until it finishes, and any deadline is the caller’s. Size one for a cold kernel cache, or warm kcache with an earlier run first.

Bench power and memory state

hipfire bench samples the GPU every 0.5 s during each measured run (--ttft and --matrix too; not --exp or --concurrency). It reads only world-readable sysfs, so it needs no root, and it never fails a bench: whatever it cannot read is reported as the string "unavailable".

• power_device (once per report): the card sampled, card (/sys/class/drm/cardN), pci, arch, selected_by and sample_period_ms. It is the daemon’s logical device 0: the card named by hardware.devices (HIPFIRE_DEVICES), else the first entry of an inherited ROCR_VISIBLE_DEVICES, else the first ROCr agent of the daemon’s arch. With --tp above 1 only that first card is sampled. If no card resolves, it holds status: "unavailable" and a reason.
• power: one object per measured run, in the order of the samples arrays (top level for the standard and --ttft paths, per row for --matrix):

gpu_metrics values are the SMU’s time-filtered averages, so the first second of a run lags the load.

When any limiter’s residency exceeds 10 % of a measured run, the bench prints one warning: line naming the worst residency per limiter, and adds it to the JSON warnings. The number is then power- or thermal-limited and is comparable only with runs whose power fields match.

High-TDP APU thermal note. On a high-TDP Strix Halo box at stock power limits, long prefill (e.g. pp8192) can be limited first by fast PPT and then by GPU temperature, which reaches the thermal point within seconds of prefill start. Capping the four SMU limits (STAPM, fast PPT, slow PPT, APU slow PPT) to a common value around 130 W (ryzenadj --stapm-limit=130000 --fast-limit=130000 --slow-limit=130000 --apu-slow-limit=130000, or the BIOS power mode) can keep the GPU below its thermal limit at some cost in burst prefill throughput; lower caps trade more throughput for energy efficiency. The effect depends on chassis, cooling and firmware, so measure the trade-off on your own machine. The power limits need root and are not visible to the bench, so record them with the result.

hipfire doctor (runtime preflight)

hipfire doctor with no --npu checks the driver, ROCm and hipfire install of this machine before any model loads, and prints one fix: line for every failed or warned check. It is GPU-free (no HIP call, no daemon start) and needs no network. It is Linux only. The only flag is -j/--json; --yes, --apply, --soak, --hours, --out and --soak-report belong to --npu and are errors without it.

Each check prints PASS, FAIL, WARN or INFO with what it found. A FAIL blocks: the last line is result: blocked (N failed) and the exit status is 1. WARN and INFO never block (result: ok, exit 0). Like the NPU report, nothing printed carries a hostname, user name, filesystem path or PCI address; GPUs are named GPU<i> gfxNNNN (1002:<device id>), with <i> the position in PCI address order. hipfire fit runs the same checks first.

Checks that need a ROCm root or a KFD GPU (hip_runtime, rocm_isa, rocm_version_floor, render_node, kernel_pack, kho_scratch, firmware) are omitted when there is none; the failing rocm_root, kfd or kfd_gpus check says why.

-j/--json prints one object, schema hipfire-doctor-runtime-1 (same privacy policy), and the exit status is the same:

{
  "schema": "hipfire-doctor-runtime-1",
  "hipfire": {"version": "0.4.3", "commit": "<first 12 hex>"},
  "result": "ok" | "blocked",
  "host": {
    "kernel": "<osrelease>",
    "amdgpu": {"source": "in-tree" | "dkms", "version": "<dkms version>" | null},
    "rocm": {"version": "<.info/version>" | null},
    "gpus": [{"index": 0, "arch": "gfx1201" | null, "pci_id": "1002:7551",
              "driver": "amdgpu" | null, "firmware": {"mes": "0x86"}}]
  },
  "checks": [{"id": "kfd", "status": "pass" | "fail" | "warn" | "info",
              "detail": "...", "fix": "..." }]
}

fix is present only on fail and warn checks; several fixes of one check are separated by newlines.

hipfire doctor --npu

hipfire doctor --npu answers one question for a Strix Halo host: can the NPU trunk prefill route run here, and does npu_trunk = auto beat the GPU-only route on this machine? Pass --npu to select it (plain hipfire doctor is the runtime preflight); the flags are:

The command is Linux only. It reads Linux sysfs, proc and device nodes, so on another OS the first checks fail and the report is blocked. The checks open neither the GPU nor the NPU; only the calibration and the soak do.

Checks. Every check prints PASS, FAIL, WARN or INFO with what it found. A failed check blocks: nothing is run and the exit status is 1. WARN and INFO checks never block and never change the tier. A failed or warned check prints a fix: line.

Tiers and profiles. The tier follows from the checks and a table of validated host profiles compiled into the CLI:

A profile row holds these fields. The report compares each against the host:

The kernel set is pack:<tag>@<commit12> when the install holds a kernel pack with a recorded <arch>.manifest.json for gfx1151 (written by hipfire kernel-pack install), or jit:<commit12> from the third word of hipfire-daemon --version when it does not; commit12 is the first 12 hex digits of the commit. It is unknown when neither is available. A row lists the kernel sets validated on it: an entry pack:<tag> matches that release’s pack at any commit (a release commit cannot name its own hash), and any other entry matches only itself. A kernel set that no row lists makes the host compatible-unvalidated by itself, even when every other field matches; JIT kernels (jit:<commit12>) are never the reference on hipx, which is pinned to the v0.4.3 release pack. The profile id and differences are in the report as profile and profile_diff.

Confirmation and exit status. After the checks the command prints what it is about to do, then asks Run it now? [y/N] on a terminal; only y or yes (any case) runs it. --yes skips the question. Without a terminal on stdin and without --yes it refuses, runs nothing and exits 2. Answering no exits 0 and still prints the report. Exit status: 0 ok, 1 blocked, a failed calibration or a failed soak, 2 refused. With --json, stdout carries only the report; the plan, the prompt, the progress lines, the soak’s request lines and summary, and the --apply note go to stderr.

Calibration. The plan it prints before asking is:

one daemon with npu_trunk=auto alternates GPU-only and auto prefills at 16K and 32K rows, 2 reps each, on prompts that never hit the prompt cache
then npu_trunk=on and npu_trunk=emulate daemons each prefill the same prompt and generate 16 tokens greedily; the two outputs must match
about 6–7 minutes on a Strix Halo (three model loads plus about 2 minutes of prefills)
uses the GPU and the NPU
changes no setting and writes no file

• A/B. One warm npu_trunk=auto daemon takes an unmeasured warm-up pair (GPU-only, then auto, 16384 rows), then two passes over 16384 and 32768 rows. Each size runs once on the GPU-only route and once on auto per pass, in alternating order (GPU first on the first pass, auto first on the second), so clock and thermal drift reaches both arms alike. The GPU-only arm is the diagnostic route (below); the daemon’s npu_trunk=auto setting and its fault latch are not changed. Every prompt is synthetic text with a different leading line, so no request is served from the prompt cache; each generates one token, and the time is the time to first token. A request that was partly served from the cache, a GPU-only request that did not log the diagnostic route line, an auto request that did, or a prompt shorter than 9000 rows fails the calibration.
• Per size, from the two samples of each arm: saved % = (median off − median auto) / median off × 100, so a loss is negative; spread % is the larger of the two arms’ (max − min) / median × 100. The verdict is gain when saved % is above the spread, loss when it is below minus the spread, and no measurable gain otherwise. The rows shown are the median of the prompt tokens the daemon reported for that size.
• Also reported when auto armed: the trunk route, the rate scales the estimator reached (gpu_x, npu_x, frozen, generation), the NPU wait per pair (median and max over the auto requests’ pairs) and the number of guard trips. When auto did not arm (for example the memlock limit is short), the report quotes the daemon’s reason and runs no A/B and no exactness check.
• Exactness. A daemon with npu_trunk=on and one with npu_trunk=emulate each prefill the same 9319-row prompt (one 8192-row chunk plus a 1127-row tail, the smallest prompt that forms an NPU pair) and generate 16 tokens greedily. exact means the two outputs are identical; MISMATCH names the index of the first token that differs; not exercised means on ran no NPU pair, so the comparison proves nothing. The daemons’ own stderr is not echoed; only their npu-trunk: lines are used.

Recommendation. The report ends with one recommendation, computed from the calibration and the effective npu_trunk:

--apply. Only the set off recommendation is applied, by the same write as hipfire config set npu_trunk off; the report then says applied: npu_trunk is now off (applied: true in JSON). In every other case it prints --apply: no change with the reason, including when the host is blocked, the question was declined or refused, or the calibration failed. It never sets auto; the set auto recommendation prints its command for you to run. A MISMATCH means the route’s on and emulate outputs disagree, which is a bug; attach the --json report to the bug report.

--json. One object, schema hipfire-doctor-npu-1. It carries no hostname, user name, filesystem path or PCI address; check details and the model tag (set only for registry models) are written without them. Top-level fields:

--soak. hipfire doctor --npu --soak replaces the calibration with a long run of the route under load, to find stability faults that the minutes-long A/B cannot. It runs the same checks, prints its plan, asks the same question, and then makes no recommendation. One npu_trunk=auto daemon (max_seq 65536) takes a loop of cold prompts of 9000, 16000, 32000 and 57000 rows, one token each, with 20 s idle between bursts so the clocks ramp each time. The run ends after --hours, or earlier at the first of:

• a fatal kernel log line, checked between requests and every second while idle;
• a request that does not complete within 1200 s (the daemon is killed), a failed request, or the daemon exiting;
• MemAvailable below 4 GiB;
• the kernel-log follower exiting (the soak then cannot watch the kernel).

The kernel log comes from journalctl -k -f, or from sudo -n dmesg -w -T --since now when the journal is not readable; with neither the soak does not start (add yourself to the systemd-journal group, or allow passwordless sudo dmesg). A line is fatal when it is an amdgpu line with page fault, timeout, GPU reset, device lost, MES failed, sq_intr, Fence fallback or a ring reset; any other amdxdna line; or a BUG:, Oops, hung_task, soft lockup, hard LOCKUP, Out of memory, watchdog: or RCU stall line. The two amdxdna messages amdxdna_insert_pages: Failed to mmap dma buf -22 and amdxdna_gem_obj_mmap: Failed insert pages, ret -22 (hipfire 0.4.2.x on Linux 7.1 or later, see the changelog) are counted as benign and never stop the run.

A heartbeat line (<UTC time> load=<1-minute load> memavail_kb=<kB>) is written and fsync’d every 10 s, so after a freeze the last line shows when the machine stopped. The soak writes only into --out (which must be a new or empty directory) and changes no setting. It exits 0 on PASS and 1 on FAIL.

After a freeze and reboot, hipfire doctor --npu --soak-report DIR reads the start time from the first line of DIR/heartbeat.log, collects the previous boot’s kernel log since then (journalctl -k -b -1; run it as a user that can read that journal, which a volatile journal does not keep), and writes it to DIR/prev-boot-kmsg.log and a summary to DIR/report.txt. The summary, also printed, gives the last heartbeat, the last request, the kernel-line counts, up to 20 fatal lines and the last 40 kernel lines. It exits 0 and does not run the checks; it fails when DIR has no heartbeat.log.

hipfire calibrate

hipfire calibrate measures the rates the engine’s load-time selectors otherwise probe for, once, and stores them in config.toml under [calibration] next to a fingerprint of this host. It never prompts and makes no network calls; everything it reads and writes is local. With no flag it measures; the flags are:

--show, --check and --clear are mutually exclusive.

What it detects. From sysfs, proc and the KFD topology, without running a kernel: per GPU, the architecture, VRAM size, PCI address (BDF) and the PCIe link speed and width (the narrowest link on the path from the root port to the GPU); per host, the CPU model, logical and physical core counts, the ISA flags (AVX2, FMA, F16C, AVX-512 variants), total RAM, and whether each GPU is unified-memory (an APU) or discrete.

What it measures. The measuring run covers, in order:

1. CPU read bandwidth on the pinned splice pool (one worker per physical core), the pool the CPU-executed offload engine runs on.
2. Per HIP-visible GPU: the pinned host-to-device copy rate (the qwen4 host-link probe itself); the zero-copy read rate, a GPU kernel reading host-mapped memory over the link; and the offload_split engine-rate probe, a CPU GEMV against a GPU GEMV over the same host-mapped bytes, for each CPU-decodable dense format of the hipfire offload-bench matrix (pool = "global"). The D2 measured default is not recorded: it needs the loaded model’s real CPU expert arm, so the first eligible not-yet-measured shape in a process is measured live at load and later matching loads in that process reuse the successful measurement (see below). A unified-memory GPU has no link to probe. A GPU the kernel lists but HIP cannot open in this run is recorded as unmeasured.

The whole run is bounded at 60 s. An optional probe that would start past the deadline is skipped and reported, and the record is written with what completed. The run ends by printing the summary, the routes the record selects and calibrated in X.X s. On hipx (7900 XTX on PCIe 4.0 x4, 16-core Strix Halo host) the whole run took 6.5 s.

Refuses while a daemon runs. The probes need an idle GPU, so the command exits with an error while a hipfire daemon is live (the same check hipfire offload-bench makes). Stop the daemon first.

How the engine uses the record. At a qwen4 model load on a discrete GPU, a fresh record replaces the load-time copy probe. The host-link class, and with it the G2 expert-staging default, the 8192-row narrow-link prefill chunk and the stage minimum rows, is taken from the stored copy rate, and the load log says [calibrated <time>]. A calibrated load still sets up the probe’s copy path with one 4 KiB copy, so it sees the same free VRAM as a probed load and places the same expert layers. A stale record (a different GPU, PCIe link, RAM, CPU or probe version) or a missing one makes the engine probe at load exactly as before, and the log names the reason. Developer env overrides still win. Run hipfire calibrate --show to see the stored values, freshness and chosen routes. The stored schema and staleness rules are in CONFIG.md.

D2 CPU split (Flash-Next, unset memory.offload_exec). The measured D2 default does not read the record. It predicts the per-layer decode cost of D2 (the production CPU arm timed on the model’s real host-resident experts, plus the current copy-transport handoff of 806.1 µs) against PCIe (bytes over a live zero-copy link rate, plus a launch floor), from 5 warmed samples of each engine, pessimistically for D2: the slowest CPU arm sample against the fastest link sample and the minimum floor. It arms D2 only when that worst case is still at least 15% faster than that best case, however wide the spread; a worst case that does not clear, a noisy inconclusive run and a failed or zero sample stay on PCIe (there is no spread threshold, the medians and spreads are logged). A stored CPU or link rate alone can never enable it: the record’s engine rates are dense, best-of figures with no spread, so they say nothing about this model’s arm or this host’s noise. The first eligible, not-yet-measured shape (device, resident codecs, expert shape) in a process gets live evidence; later loads of the same shape in that process reuse the successful measurement (the log says so), and failures are not cached. hipfire calibrate accordingly no longer writes pool = "splice" engine entries, and an entry an earlier version wrote is ignored. The load logs one line with both decision costs, their medians, the inputs, the sample spreads and the probe’s own wall cost: qwen4 D2 default (N host split layers): ON|off (live samples (…): worst-case D2 … us vs best-case PCIe … us …). The explicit memory.offload_exec, developer.qwen4_d2_default, HIPFIRE_MOE_CPU_EXPERTS=0, a load with no host split layers and a unified-memory GPU decide before any probe. The probe is a few milliseconds of CPU and 5 short link reads, resolves after the load’s free-VRAM sizing reads and frees its device buffers, so it does not change expert placement; it does not pause the PLE prewarm (contention can only turn D2 off). Rules: CONFIG.md.

D2 line in --show. For each device whose entry is fresh, --show adds one line, D2 default decided at model load …: the decision needs the loaded model’s real CPU expert arm and live link samples, which a stored record cannot supply, so --show states that and never prints an ON or off verdict it cannot make (off (unified memory: no link to beat) on a unified-memory GPU, which never probes). It is produced by the same d2_default::decide the load uses, with no model arm to measure. A stale, missing or unmeasured device has no D2 line, only the freshness reason. With --json each device carries d2_default: null or an object {status, text}, where status is load (decided at model load) or off (unified memory). Fields and rules: CONFIG.md.

hipfire fit

hipfire fit <model> [--ctx N] [--no-fetch] [-j|--json] [-v|--verbose] answers, before you pull or load, whether a model runs on this machine and how. It reuses the engine’s own admission and placement code (hipfire_loader::admission and, for Flash-Next, hipfire_arch_qwen4::expert_residency) with the same load parameters hipfire run would build from your config; it has no sizing formula of its own. It opens no GPU, never loads the HIP runtime and starts no daemon (the runtime preflight only asks hipfire-daemon --version). When no device selection is configured or inherited, fit names every KFD GPU in ROCR_VISIBLE_DEVICES in its own process, so admission’s device count is the one the daemon’s HIP would report without fit loading HIP.

<model> is an installed tag, alias, file name or path, or a registry tag that is not installed. Nothing is pulled.

What it reads. The model file’s header (installed file, or for a tag that is not installed a sparse replica fetched by HTTP Range under <HIPFIRE_HOME>/cache/fit-headers/: the header, metadata and tensor index, plus the few payload records admission itself reads; about 14 to 37 MB per model, and for Flash-Next on a GPU whose PLE rows are q4 (gfx1151) also the q4 table’s embedded rank section, about 305 MiB, because admission verifies it. Replicas are reused from the cache without network), your resolved config and per-model overrides, the KFD topology and sysfs for each GPU (architecture, VRAM total and used, PCIe link), /proc/meminfo, the TTM pool and GTT limit, the hipfire calibrate record when it is fresh, and the bundled receipts table registry/receipts-v1.json. Free device memory is computed the way the engine’s device_mem_info reports it, without HIP (discrete: VRAM total minus used; unified-memory APU: the smaller of the GTT headroom and available RAM less CMA plus the TTM pool). Once a process opens the GPU, HIP reports slightly less; the report says so in its notes.

Plan lines. topology (single GPU, or pipeline/expert parallel across the devices your config selects), kv (backend and mode), context (the number admitted and its bound: user, config, or automatic, with the trained window), capacity (for Qwen3.5/3.6 the admission’s own projection of the largest context the card holds, with the free memory, weights, prefill scratch and rounding terms and the bytes per token), placement (fully resident, or how many Flash-Next expert layers stay in VRAM and how many bytes are host-mapped), decode (the route the params select: ar, mtp, dflash) and routes (the host-link route choices and why, when a calibration record is fresh).

Findings. Each is FITS, UNDETERMINED or DOES NOT FIT, with a fix: line when it fails, and when it is undetermined and there is something to change. Ids: preflight (the hipfire doctor runtime checks; Linux only), model (the file was found, or fetched), vram (a GPU is present and has room), admission (the engine’s own admission accepted or refused the load, for example an unsupported KV backend on this platform), weights, context (the context fits the card), experts (Flash-Next expert placement: how many expert layers stay resident), host_ram and gtt (Flash-Next host-mapped experts against available RAM and the GTT cap).

Verdict and exit status. The verdict is the fold of every finding: DOES NOT FIT if any finding does not, else UNDETERMINED if any is, else FITS. Exit status 0 is FITS, 1 is DOES NOT FIT (including a blocked runtime preflight), 2 is UNDETERMINED.

Decided at load. Terms only the GPU can size at load are not guessed. Flash-Next’s load reserve needs a live device for some terms, so fit uses a sound lower bound: the resident-expert count it reports is an upper bound and the host-mapped bytes a lower bound; a host gate that already fails on that bound is a definite DOES NOT FIT, and a placement that depends on the omitted terms is reported as decided at load. Multi-GPU (pipeline or expert) capacity is sized per rank at load (UNDETERMINED, with each device’s free memory listed in the notes), and an architecture with no pre-load KV projection reports its context as sized at load. The report assumes the HIP virtual memory manager is available (ROCm on native Linux provides it); admission still applies its own platform refusals.

Quant tiers. When the tag is a family root with more than one tier, every tier is evaluated the same way and a table (tag, size, verdict, capacity or placement) is printed with largest tier that fits at <ctx>: <tag>. A tier’s verdict is the model’s own fit; machine-level blockers (the runtime preflight, device visibility) are reported once above and apply to every tier. hipfire pull <family> still defaults to the root tier. The requested model’s resolved process configuration (the settings that apply to the whole process, as hipfire run installs them) is installed once, before the machine facts and admission are computed, and every tier is evaluated under that one process policy. A tier whose own resolved process settings differ from it is still given its verdict, but its summary says per-model settings differ: run hipfire fit <tier> for its exact answer.

Expected speed. Receipts for the same tag, GPU architecture and topology, and whose placement is the plan’s placement (resident or host-experts; a receipt whose source does not state its placement never matches), each with its tokens, route, date, class (fast, interactive, usable, slow, from fixed tok/s bands) and source. Rows marked historical (an older engine, or an artifact the registry no longer serves, such as the Flash-Next numbers measured on the retired monolithic file) stay in the table but are never applied. A tag with no matching receipt prints no published receipt for <tag> on <arch> with this placement; measure with hipfire bench <tag>. Receipts are never carried across tag, architecture, topology or a placement the source states and the plan does not have.

JSON. Schema hipfire-fit-1: schema, hipfire (version, commit), input, model (tag, file, installed, replica, size_bytes, arch_id, quant), machine (devices, RAM, GTT, link facts), plan (device_index, topology, kv_backend, kv_backend_reason, kv_mode, max_seq, bound, model_ctx, projection, placement, decode_route, routes), findings (id, verdict, detail, fix), verdict, tiers, receipts, notes, details. Like the doctor report it carries no hostname, user name, home path or PCI address; the model is named by file name only.

Where files live

Override catalog path with HIPFIRE_MODELS_CATALOG_PATH if needed.

Dynamic registry

1. Bundled registry/v1.json (embedded in the binary).
2. Optional network fetch of registry/v1.json → 24h cache.
3. Invalid remote payloads are rejected wholesale. hipfire diag reports which source the process used. Entry metadata may include size, arch_id (architecture-ids.md), quant, and hashes — registry presence is not a runtime admission (INDEX.md, admissions.yml).

Useful environment overrides

Single-invocation knobs (non-exhaustive; full list in env-vars.md):

Errors and next steps