# Qwen3.8-Flash-Next on an Apple M5 (32 GB), Transformers expert streaming

Following Strata's Mac protocol (200-token story decode, ~4K-token prompt prefill), against its published
M4 Mac mini 32 GB numbers (Q2_0: 13.5 tok/s decode, 93.6 tok/s prefill; IQ3_XXS: 9.2 tok/s decode).

- `convert_fp8.py`: one-time conversion from `Qwen/Qwen3.8-Flash-Next-FP8`, shard by shard and resumable: dense
  weights to BF16 (9.2 GB), experts to 3-bit MLX affine, group 64 (49 GB, 2.15 MB per expert), n-gram table kept FP8
  (48 GB) for `disk_embeddings`. About 5 hours on this Mac, mostly downloading.
- `bench.py`: the benchmark; `run_guarded.sh` runs it under a watchdog that kills it on critical memory pressure or
  +2 GB of swap.

| Report | Cache | Expert reads | Decode tok/s | Prefill tok/s (4,059 tokens) |
|---|---|---|---|---|
| `story-2g.json` | 2 GiB | page cache | 3.69 | – |
| `story-3g.json` | 3 GiB | page cache | 3.95 | – |
| `full-3g-pagecache.jsonl` | 3 GiB | page cache | 4.10, 4.01 | 70.5 (2nd run stopped by the watchdog) |
| `full-3g-bypass.json` | 3 GiB | bypass | 3.85 | 73.5 |
| `full-4g-pc.json` | 4 GiB | page cache | 4.26, 4.22 | 71.7, 77.3 |
| `story-4g-io8.json` (8 I/O threads) | 4 GiB | page cache | 4.74 | – |
| `story-6g.json` | 6 GiB | page cache | 4.74 | – |
| `story-6g-io8.json` (8 I/O threads) | 6 GiB | page cache | 4.77 | – |
| `full-8g-io8-pagecache.jsonl` (8 I/O threads) | 8 GiB | page cache | 4.79, 5.01 | stopped (critical memory pressure) |

The 4-8 GiB runs were made after closing other apps (swap 6.6 GB instead of 10-16 GB).

`decode_breakdown_no_experts.txt`: decode with the expert computation replaced by zeros (8.4 tok/s), and per-module
time with a sync after each module: the gated linear-attention layers (PyTorch reference on MPS) dominate.

Device memory: 11.2 GiB after loading with a 2 GiB cache (9.2 GB dense + pool); a 4K prefill peaks ~6 GiB above
that. A 4 GiB cache drove macOS to critical memory pressure on this machine (10+ GB of swap already in use by other
apps).

`prefill_path_check.txt`: on 400 tokens of real text, the grouped sorted `gather_qmm` prefill (float32) vs the
unsorted path (bfloat16): same perplexity and top-1 tokens; per-layer MoE outputs differ by 0.8% at layer 0 (bf16
rounding), growing with depth.

## Metal gated-delta-rule kernel and quantized dense layers (later runs)

| Report | Dense | Cache | I/O threads | Decode tok/s | Prefill tok/s |
|---|---|---|---|---|---|
| `ab-3g-io8-mla0.json` (PyTorch decode step) | BF16 | 3 GiB | 8 | 3.91, 4.02 | – |
| `ab-3g-io8-metal.json` (Metal decode step) | BF16 | 3 GiB | 8 | 4.69, 4.59 | – |
| `full-4g-io8-metal.json` | BF16 | 4 GiB | 8 | 4.96, 4.92 | 72.7, 79.9 |
| `full-8g-io8-q8dense.json` | 8-bit (MetalConfig) | 8 GiB | 8 | 6.19, 6.29 | 69.0, 73.8 |
| `full-8g-io8-q4dense.json` | 4-bit (MetalConfig) | 8 GiB | 8 | 6.41, 6.43 | 77.1, 80.1 |

`dense_quality.txt`: perplexity on 1,024 tokens and top-1 agreement with BF16 dense. `quantized_matmul_bf16_bug.py`:
the M5 bf16 bug in the dense quantized matmul. `ssd_read.py`: uncached SSD read throughput (7.1 GB/s).

## Decode on four different prompts, and cache-policy replay (2026-11-14)

`bench.py` now measures decode on four different prompts, once each, starting from a cold cache: repeating the story
replayed the same routing on a warm cache (91% hits, 8.5–8.7 tok/s in `story-12g-current.json`).

| Report | Dense | Cache | Decode tok/s | Prefill tok/s |
|---|---|---|---|---|
| `decode-4prompts-12g.json` (story, Python, history, French) | 8-bit | 12 GiB | 8.74, 8.77, 8.09, 8.10 (median 8.42); hits 89.3, 89.8, 86.8, 86.8% | – |
| `prefill-cold-10g.json` (1K chunks, cold cache) | 8-bit | 10 GiB | – | 71.8 |

Other apps closed; with Firefox open the same runs gave 8.26, 7.83, 7.64, 7.56 and 65.6. `decode_profile.py` /
`decode_profile.json`: host-side timers on the expert path (routing sync, read waits, bookkeeping) per decoded token.

`cache_policy_trace.py` logs the experts of every MoE call over five generations (`routing_trace_5prompts.json.gz`);
`cache_policy_sim.py` replays them through LRU, LFU, warm starts, pinned hot experts and Belady's optimum
(`cache_policy_replay.txt`). At 12 GiB the optimum is 3–5 points above LRU and equals the first-use floor for the
story; warming up gains ~1 point and pinning loses 2–15.

## Decode speedups: shared-expert overlap, early MPS commits, fused hyper-connection projections (2026-11-14)

| Report | Decode tok/s (story, Python, history, French) | Run median |
|---|---|---|
| `decode-4prompts-12g.json` | 10.66, 10.85, 9.54, 9.43 | 10.10 |
| `decode-4prompts-12g-run2.json` | 10.72, 10.72, 9.49, 9.32 | 10.10 |
| `decode-4prompts-12g-run3.json` | 10.75, 10.00, 9.49, 9.76 | 9.88 |

Cold prefill (10 GiB, 1K chunks): `prefill-cold-10g.json` 67.5, `prefill-cold-10g-run2.json` 69.2 tok/s (early commits on
or off: 69.2 / 69.8, no effect on prefill). `dense_ablation.py` / `dense_ablation.txt`: dense-path cost per module
category (experts zeroed, replay ablation). `decode_ppl.py`: teacher-forced perplexity through the decode path.
`mps_commit_repro.py`: the MPS command-buffer commit issue.