← Back to the head-to-head and the gap analysis
Details: Qwen3.8-Flash-Next on a 32 GB Mac with Transformers
Experts streamed from disk through a bounded device cache · Apple M5 MacBook Pro, 32 GB unified memory, MPS · torch 2.15 · compared with Strata 0.1.60 run on the same Mac
Strata runs Qwen3.8-Flash-Next
on a 32 GB Mac by streaming its experts from the SSD, since they are far too large to fit in memory. This page is a
proof of concept that does the same with Transformers: from_pretrained with a new
experts_offload option, running on PyTorch MPS instead of a dedicated engine.
Updated 2026-11-17. Earlier sections keep their dated measurements (3-bit experts, 8-bit dense layers) as history; the current state is in the first tables, on the main page and in docs/kernels.md (every kernel, and which weights come from safetensors or from the GGUF).
Upstreaming expert streaming as from_pretrained(..., experts_offload=...):
huggingface/transformers#50717 ·
RFC. The Qwen3.8-specific parts (disk-backed n-gram table, Metal decode kernels), the
converter and every benchmark script live in
tarekziade/moe-expert-streaming, a package on top
of that PR.
Interactive architecture requires JavaScript. See the Architecture section below for the full model, storage and kernel details.
Results
Same protocol as Strata's Mac guide, decode tok/s on 200-token answers (greedy) and prefill tok/s on a 4,059-token prompt, OS page cache on, with one change: decode runs four different prompts once each (a story, Python code, a history explanation, French grammar), starting from a cold cache, instead of repeating the story. The expert cache carries over from one generation to the next, so a repeated prompt replays the same routing on a warm cache (91% hits instead of 89% for the story), and the story is among the fastest of the four: 10.7 tok/s, against 9.4–9.5 for the history and French answers, whose hit rate is 87% (3-bit experts, 2026-11-14). Other apps closed (an open browser costs ~0.7 tok/s on this Mac). Strata was installed and run on this same Mac with the same prompts (same tokenizer, same 200-token answers, the same 4,059- and 6,062-token prompts), thinking off, greedy, no MTP, no speculative drafts (0 proposed), no prompt reuse, a 12 GB expert arena and its one-read-per-expert copy of the experts, three rounds of the whole protocol (report, client). Its published M4 Mac mini numbers are kept for reference.
| Engine | Experts | Dense layers | Expert cache | Decode tok/s | Prefill tok/s (4K) | Perplexity |
|---|---|---|---|---|---|---|
| Strata 0.1.60 (this M5) | IQ3_XXS (3.06 bits/weight), closest to our 3-bit experts | pack quant | 12 GB | 16.3, 15.0, 14.4, 14.7 median 14.8 (3 rounds) | 204, 221, 220 median 220 | 4.119 vs 4.052 for Q8_0 (+1.6%) ‡ |
| Strata (published, M4 Mac mini 32 GB) | IQ3_XXS | pack quant | 11.5–14.7 GB | 9.2 | 65.8 | same build |
| Strata (published, M4 Mac mini 32 GB) | Q2_0 | pack quant | 11.5–14.7 GB | 13.5 | 93.6 | 4.475 vs 4.052 for Q8_0 (+10.4%) ‡ |
| Transformers (M5), 2026-11-17recommended | Strata's IQ3_XXS experts from its .experts file (IQ2/IQ3 gate/up, IQ4_NL or Q2_0 down, 1.75 MB each) | Strata's GGUF tensors (Q4_K–Q6_K, IQ4_XS, IQ3_S) + 8-bit | 12 GiB (10 for the 4K prefill) | medians 13.81, 13.80 (2 runs of the 4 prompts) | 212.2, 225.6 (2K chunks) | 4.984 § |
| Transformers (M5), 2026-11-16 | 3-bit | same GGUF tensors | 12 GiB (10) | medians 11.59, 11.56 | 156.5, 159.5 (2K chunks) | 4.925 § |
| Transformers (M5), 2026-11-14 | 3-bit (~3.5 bits/weight with scales and biases) | 8-bit | 12 GiB | 10.7, 10.7, 9.5, 9.4 median 10.0 (3 runs) | – † | 4.901 |
| Transformers (M5), 2026-11-14 | 3-bit | 8-bit | 10 GiB | – | 67.5, 69.2 (1K chunks, cold) | 4.901 |
| Transformers (M5), earlier | 3-bit | 4-bit | 8 GiB | 6.4, 6.4 * | 77.1, 80.1 * | 5.355 (+9.3%) |
| Transformers (M5), earlier | 3-bit | BF16 | 4 GiB | 5.0, 4.9 * | 72.7, 79.9 * | 4.901 |
§ Prefill-path perplexity on 1,024 tokens of Transformers' philosophy.md (6 GiB cache), the same
text as the 4.901 measurements; with BF16 dense layers and 3-bit experts 4.9007. Kernel-to-kernel rounding moves it by
up to ±0.4% (the PyTorch Gated DeltaNet path gives 4.9964 on the same build). Through the decode path (512 tokens),
Strata's experts give 3.3116 and the 3-bit ones 3.4088, both with the GGUF dense tensors. The 2026-11-16/17 rows are
alternating A/B runs; their reports are linked from the main page.
Older rows: perplexity on 1,024 tokens of Transformers documentation with the same 3-bit experts; 8-bit dense layers pick
the same top-1 token as BF16 at 95% of positions, 4-bit at 82%. 8 I/O threads. † a 4K prefill with a 12 GiB cache needs
more free memory than this Mac had with its usual apps open (it ran at 60–70 tok/s on a quieter day); with 10 GiB it
fits. Long prompts are prefilled in 1,024-token chunks, as Strata does on Macs under 96 GB. * measured before the
read-pipeline and decode-kernel work, two runs of the same prompt (warm cache for the second). Neither Strata run uses MTP. Strata's decode rates are the third round's four prompts (the first round's story,
on a cold arena: 15.9); the server's own timings agree with the client's within 1%.
Through the decode path itself (teacher-forced, one token per step, 512 tokens), perplexity is 3.3558 with BF16 dense
layers and 3.3629 with 8-bit ones (+0.2%, same top-1 token at 94.5% of positions;
data,
script). ‡ Strata's quality table, as quoted by
ModelFit: perplexity
relative to its own Q8_0 build, on text we do not know, so it is not comparable with our absolute numbers (ours compare
dense-layer precisions with the same 3-bit experts).
Decode after a 6K-token document
Strata's third Mac row: 200 tokens decoded after reading a long document, so the attention layers see ~6K earlier tokens and the prefill has just churned the expert cache. Strata on this Mac now reads the document 1.16× faster and decodes after it 1.2× faster (3.4× and 1.9× on 2026-11-14).
| Engine | Experts | Expert cache | Prompt | Decode tok/s | Prefill tok/s |
|---|---|---|---|---|---|
| Strata 0.1.60 (this M5) | IQ3_XXS | 12 GB | 6,062 tokens, 1K chunks | 13.6, 13.7, 13.8 median 13.7 | 222 |
| Strata (published, M4 Mac mini 32 GB) | IQ3_XXS | 11.5–14.7 GB | 6K document | 7.7 | – |
| Strata (published, M4 Mac mini 32 GB) | Q2_0 | 11.5–14.7 GB | 6K document | 13.1 | – |
| Transformers (M5), 2026-11-17, GGUF dense | Strata's IQ3_XXS | 12 GiB | 6,062 tokens, 2K chunks, cold | 11.56, 11.25 | 197.5, 186.6 |
| Transformers (M5), 2026-11-16, GGUF dense | 3-bit | 12 GiB | 6,062 tokens, 2K chunks, cold | 8.75, 8.94, 8.44 | 158.2, 157.6, 157.2 |
| Transformers (M5), 2026-11-14, 8-bit dense | 3-bit | 12 GiB | 6,062 tokens, 1K chunks, cold | 7.3 (1 run) | 64.5 |
| Transformers (M5), 2026-11-14, 8-bit dense | 3-bit | 10 GiB | 6,062 tokens, 1K chunks, cold | 7.1 (1 run) | 64.5 |
2026-11-17: after the document, decode took 113 ms per token against 80 after a short prompt. Past the indexer's 2,048-token budget the model built a mask over the whole cache and ran SDPA over every key (0.95 ms per attention layer at 6K keys); a split-K flash decoding kernel now reads only the selected blocks and the tail (0.12 ms) and the indexer's decode step is two kernels (0.47 -> 0.11 ms per layer). The rest of the gap is expert reads: ~61 misses per token after the document against ~42 after a short prompt. Long prefill chunks now leave their expert installs at the LRU's cold end (-7% misses; see the cache-policy section).
2026-11-14: the document is the 4K prompt's synthetic harbour records, extended to 6K tokens; ask: summarize them. The 6K prefill peaks at 25.4 GiB of device memory with a 12 GiB cache (24.0 with 10 GiB), so it needs other apps closed. Decode drops from 10.0 to 7.3: the cache holds what the prefill touched (56% of lookups missed over the whole run; 292 GB read from the SSD) and full attention covers 6K tokens. Strata drops far less on the same Mac (14.8 to 13.7). The two cache sizes give the same answer for ~150 tokens, then different wording (the prefill groups experts differently, which changes the summation order). Reports: 12 GiB, 10 GiB.
A fair comparison
With both on the same Mac and the same prompts, and Transformers now running Strata's own IQ3_XXS experts and dense tensors, Strata is 1.08× faster at short decode, even at a 4K prefill and 1.2× faster at decode after a long document (with our 3-bit experts on 2026-11-14: 1.5×, ~3.2× and 1.9×). The gap was software. Strata publishes several expert quantizations; the one to compare with is the one closest in quality, not the fastest:
| Strata build | Same top-1 token as Q8_0 | Perplexity (Q8_0: 4.052) | Decode on the M4 mini (tok/s) | Decode on this M5 (tok/s) |
|---|---|---|---|---|
| IQ3_XXS | 84.5% | 4.119 (+1.6%) | 9.2 | 14.8 |
| Q2_0 | 80.6% | 4.475 (+10.4%) | 13.5 | not run |
- Quality: the recommended configuration runs Strata's IQ3_XXS experts themselves, so the comparison is like for like. Our own 3-bit MLX affine experts (group 64) sit between the two builds. We have not measured them against 8-bit experts (the FP8 source was deleted after conversion), so IQ3_XXS is the like-for-like row, and Q2_0 trades ~10% perplexity for its 13.5 tok/s.
- Bytes per expert: with a 16-bit scale and bias per 64 weights, our 3-bit experts take ~3.5 bits per weight (2.15 MB), against 1.75 MB on average for Strata's: more to read per miss and fewer experts per GiB of cache. Running Strata's experts read 30–40% fewer bytes per answer and gave +12–14% decode.
- Hardware: settled by running both on this M5. Strata gains more from the M5 than its ~28% extra memory bandwidth (153 vs 120 GB/s): +61% at short decode and +78% after the 6K document over its M4 numbers, helped by an SSD that reads 7.1 GB/s here (2.97 GB/s on Strata's M4, per its guide). Comparing our M5 numbers with its M4 ones made Transformers look on par; it is not.
- Protocol: identical prompts for both. Since 2026-11-16 ours are alternating A/B runs of the previous and the new build (other apps closed), Strata's medians of three rounds (its spread is small: 13.6–13.8 after the 6K document).
- Speculative decoding: Strata leaves MTP off on 32 GB Macs ("MTP was slower"): with streaming, draft tokens touch more experts. On a 128 GB M5 Max it takes Q2_0 from 87.3 to 108.2 tok/s.
The two Macs
| Machine | Chip | CPU | GPU | Memory bandwidth | Unified memory | SSD |
|---|---|---|---|---|---|---|
| Strata's (Mac mini) | M4 | 10 cores (4P + 6E) | 10 cores | 120 GB/s | 32 GB | size not stated |
| This one (MacBook Pro 14") | M5 | 10 cores (4P + 6E) | 10 cores, Neural Accelerators in each | 153 GB/s | 32 GB | 1 TB, 7.1 GB/s measured |
Apple's launch specs. Strata's guide says "M4 Mac mini 32 GB", which is the base M4 (the M4 Pro Mac mini comes with 24, 48 or 64 GB). The M5 has ~28% more memory bandwidth and its SSD reads 7.1 GB/s even for the 2.15 MB random reads experts need, so numbers measured on the two Macs are not directly comparable; the comparison above uses Strata run on this M5.
Architecture
Model: Qwen3.8-Flash-Next, 48 decoder layers, hidden size 2,560, vocabulary 248,320. Every layer has a token mixer and an MoE block, each wrapped in a 4-stream hyper-connection. Token mixers: 36 Gated DeltaNet (linear attention) layers and 12 full-attention layers (one every 4), whose attention is sparse: an indexer keeps at most 2,048 earlier tokens per query. MoE: 512 routed experts (top-10, intermediate size 640) plus one shared expert per layer. Layer 1 adds a per-layer n-gram embedding (320M rows × 160). MTP and vision are not loaded.
Where every weight lives
| Weights | Size | Stored as | Where |
|---|---|---|---|
| Routed experts (48 × 512), recommended | ~43 GB | Strata's .experts file: its GGUF blocks (IQ2/IQ3 gate/up, IQ4_NL or Q2_0 down), 1.36–2.33 MB per expert by layer, one read each | SSD; 12 GiB in 10 device pools, one per expert size (LRU each) |
| Routed experts, 3-bit option | 49 GB | 3-bit MLX affine, group 64, 2.15 MB per expert | SSD; 12 GiB in one device slot pool |
| N-gram embedding table | 51 GB | FP8 (as in the checkpoint) | SSD, read row by row (a few rows per token) |
Large dense projections (Gated DeltaNet qkv/z/out, attention q/k/v/o, shared experts, lm_head), recommended | ~3.9 GB | Strata's GGUF tensors as stored (Q4_K–Q6_K, IQ4_XS, IQ3_S); 8 tensors in Q2_0, which ggml's Metal kernels cannot multiply, stay 8-bit | device |
| Other dense layers (Gated DeltaNet a/b, hyper-connection projections, indexer), and all of them in the 8-bit option | ~5.3 GB in the 8-bit option | 8-bit MLX affine (MetalConfig) | device |
| Embeddings, norms, routers, conv weights, small gates | ~1.3 GB | BF16 | device |
Converted once from Qwen/Qwen3.8-Flash-Next-FP8
by convert_fp8.py (dense to BF16, experts to 3-bit, n-gram table
kept FP8); MetalConfig quantizes the dense layers at load time. The recommended configuration then swaps in
Strata's GGUF (gguf_dense=, gguf_experts=True); the converted checkpoint still provides the
small layers, the n-gram table and the expert layout.
Kernels (recommended configuration)
Every kernel, the switches that turn each off, and which weights come from safetensors or from the GGUF:
docs/kernels.md.
Host code is pure Python; device work is Hub kernels or Metal source compiled with torch.mps.compile_shader.
| Part | Decode (one token) | Prefill (2K-token chunks) |
|---|---|---|
| GGUF dense projections | fused dequantize-gemv of ggml-org/ggml-quantization | its dequantize + torch.matmul |
| Other quantized linears (8-bit) | quantized_matmul of kernels-community/mlx-quantization-metal-kernels v2 (float32 above ~30 rows on M5, see known issues) | |
| Routed experts | ggml's mul_mat_id on the size-class pools (up to 512 routes); our Q2_0 gather-gemv for the 30 layers whose down projection is Q2_0 | our tile GEMM on Metal 4's matmul2d (the M5's matrix units), dequantizing the GGUF blocks inside the kernel (ggml's dequantizers) and reading the hidden states through a row index, 32-row tiles plus 16-row tails; a combine kernel sums each token's routes |
| Gated DeltaNet | one matmul for the four input projections; one kernel for the conv state, conv, SiLU and gates; one for the delta rule and the gated RMSNorm (~34 ops → 4 dispatches, bit-exact) | conv + SiLU writing token-major rows (bit-exact); q/k L2 norm + k.q; the delta rule as a scan after Strata's gdn_scan_r (a lane per state column, the state in registers); the gated RMSNorm (sigmoid gate) as one kernel. 67 → 28 ms per 2K-token layer |
| Full attention (12 layers) | past 2,048 keys: the indexer's query norm + rotary and block scores as two kernels, topk, then split-K flash decoding over the selected blocks and the tail only; below: SDPA | the indexer's token selection vectorized (one scores matrix and one topk instead of a Python loop per query), then SDPA under its mask |
| Hyper-connections | group RMSNorm, SiLU, stream mixing: three kernels; the → 320 and → 4 projections fused into one quantized_matmul | the same three kernels for every token; separate projections (the fused one is wrong on M5 for many rows) |
| Routers, norms, embeddings, sampling | PyTorch ops on MPS; the n-gram table read from disk row by row | |
With the 3-bit option, routed experts run through gather_qmm of the MLX kernel repository (sorted,
64 experts per call in prefill) and every dense layer through quantized_matmul.
Published kernels
Our Metal kernels are on the Kernel Hub,
one repository each, and the package loads them from there (pinned to commits in
kernels.lock).
They are Metal source compiled at first use with torch.mps.compile_shader (Apple silicon, PyTorch MPS):
| Repository | Used for | Kernels | License |
|---|---|---|---|
| tarekziade/gguf-mm-metal | Routed experts, prefill | tile GEMM over GGUF expert banks on Metal 4 matmul2d (blocks dequantized in the kernel, rows read through an index) + route combine | MIT (ggml, Strata) |
| tarekziade/q2-0-metal | Routed experts, decode | Q2_0 (ggml type 42) gather-gemv and dequantize | Apache-2.0 |
| tarekziade/gated-delta-net-decode-metal | Gated DeltaNet, decode | conv state + conv + SiLU + gates; delta rule + gated RMSNorm | Apache-2.0 |
| tarekziade/gdn-scan-metal | Gated DeltaNet, prefill | causal conv + SiLU, q/k L2 norm, the gated delta rule as a scan | MIT (Strata) |
| tarekziade/qsa-decode-metal | Full attention, decode | split-K flash decoding over the selected blocks and the tail; indexer query norm/rotary and block scores | Apache-2.0 |
| tarekziade/gated-rms-norm-metal | Gated DeltaNet output norm | gated RMSNorm (SiLU or sigmoid gate) | Apache-2.0 |
| tarekziade/hyper-connection-metal | Hyper-connections | group RMSNorm, SiLU, stream mixing | Apache-2.0 |
Load one with get_kernel("tarekziade/gdn-scan-metal", version=1, trust_remote_code=True)
(kernels ≥ 0.18; the namespace is not a trusted kernel publisher). Several take the Transformers module
itself (Qwen3.8's layer layout) rather than plain tensors.
Expert streaming
- Loading:
from_pretrained(..., experts_offload={...})builds the model without expert weights and loads everything else normally;quantization_config=MetalConfig(bits=8)quantizes the dense layers. - Cache: preallocated device pools of expert slots with a per-layer table expert → slot kept on the host: one pool shared by all layers for the 3-bit experts (12 GiB hold ~6,000 of the 24,576), one per expert size for Strata's (10 sizes, 153 slots per layer on average, LRU each, gate/up and down in separate contiguous pools).
- Each MoE layer: the router's choices are copied to the host (the one host synchronization per layer), misses are
read on 8 I/O threads into recycled pinned host buffers (one contiguous read per expert, through descriptors opened
once), copied to free slots without blocking, and the layer sends its slot indices in one small copy and runs
gather_qmmon the pool. - Prediction (optional,
prefetch_depth, off by default): the routers of the next layers run on the current hidden states and their guessed experts are read in the background. On the M5 the guesses cost more than they hid (decode 11.1 → 11.7 tok/s without them). - Prefill: a 2K chunk touches nearly every expert of every layer. The next layer's experts are read while the
current one computes (
prefill_readahead_routes), and the finished reads are installed behind its compute. Chunks of 512 tokens or more leave their installs at the LRU's cold end and do not refresh hits: plain LRU evicts each expert just before the next chunk needs it, and leaves decode a cache of scan leftovers.
Decode speedups since the first published numbers (2026-11-14, 3-bit experts)
Each measured as alternating A/B pairs (4 prompts per run, 12 GiB cache), commits on the POC branch. Later changes (GGUF dense layers and experts, prefill kernels, decode after a long document) are on the main page and in the package README:
| Change | Decode tok/s (before → after) | Output |
|---|---|---|
| Queue the shared expert after the routing copy (pinned buffer + event), so it runs while the host does the cache bookkeeping | 8.25 → 8.55 (+3.6%) | identical |
| Commit queued MPS work early: record an event after each decoder-layer sub-block and after the expert kernels | 8.52 → 9.40 (+10.3%) | identical |
| Hyper-connections: one 8-bit matmul for the 10240 → 320 and 10240 → 4 projections (the 4-row call alone took 49 µs, twice the 320-row one) | 9.48 → 10.03 (+5.8%) | rounding differs; decode-path perplexity 3.3642 → 3.3629 (512 tokens), top-1 agreement 96.7% |
The MPS commit issue. PyTorch's MPS stream encodes small kernels into a command buffer and commits it only at the
next synchronization (or under memory pressure). Decode is made of small kernels and synchronizes once per MoE layer, so
after each routing wait the GPU idled while the host encoded the experts and the next layer, then ran them only at the
next wait. Launching 1,000 small matmuls and then doing 106 ms of host work takes 213 ms; recording an event after the
launches (it commits without waiting) brings it to 109 ms
(reproducer). With the experts replaced by zeros, the same
fix takes decode from 67 to 56 ms per token. It would be best fixed in PyTorch; Transformers works around it with
experts_offload={"mps_commit": True} (the default on MPS).
Do these changes help other MoEs?
Two of the three decode changes are generic, one targets a family of models, one is Qwen3.8-only:
| Change | Applies to | Checked |
|---|---|---|
| Routing copy to a pinned buffer behind an event | every streamed MoE | OLMoE: identical output, never slower |
Early MPS commits (mps_commit) | every streamed MoE on MPS (hooks on each decoder layer's sub-blocks) | OLMoE: +3% when streaming, more while the cache fills; Qwen3.8: +10% |
| Shared expert queued after the routing copy | MoE blocks with gate, shared_expert and shared_expert_gate: Qwen2-MoE, Qwen3-Next, Qwen3.5-MoE, Qwen3-Omni (talker), Qwen3.8 | all five blocks have the same forward, which the override reproduces; for upstreaming it should be an explicit opt-in per model rather than a match on attribute names |
| Fused hyper-connection projections | Qwen3.8 only (its own kernel file) | – |
OLMoE-1B-7B (16 layers × 64 experts, 4-bit experts, 3.4 GB), same four prompts, 200 tokens, greedy, cold cache first; before is the branch just before the three changes (script, reports):
| Expert cache | Before (decode median tok/s) | Now, commits off | Now | Output |
|---|---|---|---|---|
| 1 GiB (streaming, 85% hits; 2 runs each) | 29.0, 32.7 | 32.7, 32.1 | 33.2, 33.8 | identical text; perplexity 8.2953 (prefill), 8.2484 (decode path) in every run |
| 4 GiB (all experts fit; 1 run each) | 60.2 | – | 73.2 | identical text |
Streaming OLMoE gains ~3%, within run-to-run noise: its dense layers are small, so the GPU had little host encoding to wait for. With a cache that holds every expert, the gain comes from the first two answers, while the cache fills (66–67 against 47–48 tok/s); once every expert is resident and no layer synchronizes, both run at the same speed (79 / 84 against 72 / 85). Unrelated to these changes: with every expert resident, prefill takes a different matmul path (perplexity 8.2144 instead of 8.2953) and one of the four answers differs from the streamed one, before and after alike.
What limited decode (2026-11-14, 3-bit experts, 8-bit dense layers)
Since then the dense projections read Strata's smaller GGUF tensors, the experts are 1.75 MB instead of 2.15, and decode went from 10.0 to ~13.7 tok/s. At that time, the dense layers, measured by ablation: experts replaced by zeros, then each module category replays its first output instead of computing, which shows what it would save if it were free (data, script). Floor: reading the category's 8-bit weights once at the M5's 153 GB/s.
| ms per token (dense part: 56 ms) | Measured | Weight-read floor |
|---|---|---|
| Gated DeltaNet (36 layers) | 21.9 | 13.6 |
| Hyper-connections (97), before the projection fusion (~5 ms less after) | 14.5 | 4.2 |
| Full attention (12 layers, with the indexer) | 6.8 | 4.0 |
| lm_head | 5.7 | 4.2 |
| Shared experts (48) | 2.6 | 1.5 |
| Routers (48) | 2.1 | 0.4 |
| PLE n-gram embedding (disk rows; its own host sync cost 10 ms before the commit fix) | 1.0 | – |
- Expert misses (10–16 ms per token before these changes): ~0.6 experts per layer and token are still being read when their layer runs; each wait is a whole read (~0.34 ms), so the guess missed them rather than being late. Most are first uses of an expert in the answer, which no cache policy avoids (see below): only better prediction of the next layers' experts or smaller experts help. Splitting reads, guessing wider (16 instead of 10 experts) and LFU did not.
- One host synchronization per MoE layer (48 per token): an MPS round trip costs ~210–250 µs even for a tiny
op, whatever the mechanism (
.cpu(), an event ortorch.mps.synchronize). - Rewriting the cache bookkeeping (pure Python trims) gained nothing measurable once the shared expert overlapped it.
- The Gated DeltaNet layers are at the memory-bandwidth limit: per layer, the merged 8-bit in-projection
(2,560 → 16,480) takes 372 µs for 44.8 MB and
out_proj142 µs for 16.7 MB, ~120 GB/s, about what the M5 sustains; the two Metal kernels add ~70 µs (data). Only fewer bytes would help: 6-bit dense layers would read 25% less, butMetalConfigsupports 2, 4 and 8 bits only, and 4-bit costs 9.3% perplexity. - Left at 8-bit: the hyper-connections (~10 ms per token, ~5 ms floor; fusing the up-projection into the mixing kernel could save a few ms) and full attention (6.8 ms, ~5 ms floor).
Would a warmup or a smarter cache policy help? Not for decode
The routing of the four decode prompts (plus the story again) was logged and replayed through a cache simulator at the 12 GiB size (5,991 expert slots), counting only the experts the model asks for (no next-layer guesses, hence a few points below the measured hit rates). Belady's policy evicts the expert needed furthest in the future: no policy can beat it.
| Simulated hit rate | Story | Python | History | French |
|---|---|---|---|---|
| LRU, cold cache (current) | 86.2% | 86.0% | 82.1% | 81.3% |
| LFU | 86.6% | 84.7% | 79.5% | 80.1% |
| LRU, warmed by the previous prompt | – | 86.9% | 82.9% | 82.5% |
| 25% of the cache pinned to the experts most used by the other prompts, LRU for the rest | 83.8% | 84.3% | 78.8% | 79.4% |
| Same, 50% pinned | 80.4% | 81.9% | 77.3% | 76.3% |
| Belady (optimal) | 89.3% | 88.9% | 87.4% | 86.8% |
One 200-token answer touches 10.7–12.6k distinct experts, about twice what the cache holds, and 10–13% of its lookups are first uses. For the story, the optimum at 12 GiB equals that first-use floor (89.3%), and LRU reaches it with a 1.5× cache. Consecutive prompts share only 51–72% of their experts, so warming up gains about one point, and pinning "popular" experts loses up to 10. New experts keep appearing through the answer (1.7 per layer and token at the start, 0.2 by token 175), so long conversations would gain a little from a better policy, but a 200-token benchmark would not. Replay output, simulator, routing trace.
After a long prefill (2026-11-17, Strata's experts, per-size pools). The same replay with the four story prompts followed by the 6K document, in misses per decode token (data): LRU 66.6 after the document and 42.5 on the stories; a static cache ranked by the stories' decode frequency, as Strata's arena is, 205 and 89 (77.7 and 48.9 with half of it static): routing has strong short-term locality, so LRU wins even on the prompts the ranking came from. What did help is the prefill: each 2K chunk cycles through nearly every expert, more than a pool holds, so LRU evicts each one just before the next chunk needs it. Leaving long-prefill installs at the LRU's cold end cut the 6K prefill's misses from 50.2K to 36.2K and the decode misses after it by 8% (measured: -7% misses, +0.6% decode).
CUDA (preview): one A10G
The same converted model also runs on an NVIDIA A10G (24 GB) through the same package (run, reports): BF16 dense layers, 3-bit experts in an 8 GiB device cache, run by a Triton kernel that unpacks the MLX format in registers (matches PyTorch at bfloat16 precision for 2-8 bits, check). The pod has 83.5 GiB of RAM and 22.8 CPUs; the experts sit on a network file system.
| A10G, same 4 prompts | Decode tok/s | Median |
|---|---|---|
| Experts read through the page cache | 2.8, 3.6, 4.5, 4.8 | 4.1 |
| All experts preloaded in a pinned host tier (as Strata keeps them in RAM) | 5.8, 5.8, 5.7, 5.8 (repeat: 5.9, 5.8, 5.8, 5.8) | 5.8 |
| Host tier filled on demand (no preload) | 2.9, 3.2, 4.1, 4.7 | 3.6 |
Quality: decode-path perplexity (teacher-forced, 512 tokens) is 3.4366 on CUDA (BF16 dense) vs 3.3558 on the Mac
(BF16 dense); the two agree on the top-1 token 95.3% of the time. The host tier
(host_cache_bytes, host_preload) is now in the PR. An answer that differed in an earlier run
came from CUDA decode not being bit-deterministic run to run, not from the host tier.
Early numbers. Decode is held back by the dense layers, which run as plain PyTorch ops on CUDA: the Metal kernels (Gated DeltaNet step, hyper-connections) have no CUDA counterparts yet, and the per-expert prefill path makes a 4K prompt slow (17 tok/s).
Known issues
- On the M5, the NAX kernels of mlx-quantization-metal-kernels v2 return wrong results for bfloat16/float16 inputs once a
call has enough rows: sorted
gather_qmmfrom 4 rows per expert (kernels-community#1332), and the densequantized_matmulfrom ~30–128 rows depending on the shape (reproducer). float32 is correct, and so is every single-row (decode) call; prefill runs those calls in float32, and the hyper-connections' fused projection is used for one token only. The GGUF expert prefill uses our own kernels and is not affected. - The first long-document run after loading has twice been killed by the memory watchdog (6K prefill with a 12 GiB cache); later runs in the same conditions pass.
- Off Apple silicon and CUDA, quantized experts use a slow PyTorch reference. On CUDA, decode uses a Triton kernel; prefill still dequantizes expert by expert.
Files: transformers.patch (the diff of huggingface/transformers#50717 at 18abece3ab, the commit every 2026-11-16/17 number uses) ·
raw reports, converter and benchmark.