# Decode after a long document, M5 32 GB, 2026-11-17

Package branch [`gguf-experts`](https://github.com/tarekziade/moe-expert-streaming/tree/gguf-experts) (`prefill-tail`
f2165ac merged), transformers #50717 at `18abece3ab`, GGUF dense layers and GGUF experts, apps closed, alternating runs.
`devB*` = new code, `devA*` = the previous `gguf-experts` (a498117). All runs: `scripts/bench.py ... --io-threads 8
--dense-bits 8 --gguf-dense ... --gguf-experts`.

Profile of the previous build (48 tokens): 113 ms per token after the 6K document vs 80 after a short prompt;
serialized per token, attention + indexer 27.8 vs 9.6 ms, the MoE block 90.2 vs 58.9 ms (expert misses 88 vs 49 per
token). New:
- QSA decode attention over the indexer's selected blocks and the tail only (split-K flash decoding): 0.95 -> 0.12 ms per
  layer at 6K keys (SDPA over the whole cache with a boolean mask before); relative error vs float32 1.6e-3 (bf16 SDPA
  2.3e-3); in the model, 48 tokens after the document: 110.4 -> 98.6 ms per token, 47 of 48 tokens identical;
- the indexer's decode step (query norm + rotary, block scores) as two kernels: 0.47 -> 0.11 ms per layer, same blocks;
- long prefill calls (>= 512 tokens) leave their expert installs at the LRU's cold end and do not refresh hits.

| Runs | Flags | New | Previous |
|---|---|---|---|
| `dev{B,A}idx-long-{1,2}.json` (after 6K doc: prefill / decode) | `--cache-gib 12 --skip-story --skip-prefill --long-doc --prefill-chunk-size 2048` | 197.5 / 11.56, 186.6 / 11.25 | 205.0 / 9.82, 193.4 / 9.73 |
| `dev{B,A}idx-decode-{1,2}.json` | `--cache-gib 12 --skip-prefill` | decode median 13.81 / 13.80 | 13.86 / 13.41 |
| `cold{512,0}-long-{1,2}.json` (new code, `GGUF_EXPERTS_COLD_PREFILL` 512 vs 0, before the fused indexer) | long doc | 201.5 / 11.27, 197.5 / 11.29; misses 40.8K, 39.9K | 199.2 / 11.14, 200.4 / 11.27; misses 43.8K, 42.4K |

`policy_sim.txt` (`gguf_policy_sim.py` on a routing trace of the 4 story prompts + the 6K document, 200 tokens each,
per-size-class pools): misses per decode token. A static cache ranked by the story prompts' decode frequency (Strata's
approach), same run: 50% of the slots + LRU 77.7 after the document / 48.9 on the stories, 75% 88.7 / 54.6, 100% 205.4 /
88.9, against 66.6 / 42.5 for LRU. Long prefill with cold-end installs: 36.2K prefill misses vs 50.2K (LRU thrashes
on each chunk's scan of nearly every expert).
