Running Qwen3.8-Flash-Next on a 32 GB Mac with Transformers

Experts streamed from disk through a bounded device cache · Apple M5 MacBook Pro, 32 GB unified memory, MPS · torch 2.15 · compared with Strata's published Mac numbers

Strata runs Qwen3.8-Flash-Next on a 32 GB Mac by streaming its experts from the SSD, since they are far too large to fit in memory. This page is a proof of concept that does the same with Transformers: from_pretrained with a new experts_offload option, running on PyTorch MPS instead of a dedicated engine.

Upstreaming expert streaming as from_pretrained(..., experts_offload=...): huggingface/transformers#50717 · RFC. The Qwen3.8-specific parts (disk-backed n-gram table, Metal decode kernels), the converter and every benchmark script live in tarekziade/moe-expert-streaming, a package on top of that PR.

Results

Same protocol as Strata's Mac guide, decode tok/s on 200-token answers (greedy) and prefill tok/s on a 4,059-token prompt, OS page cache on, with one change: decode runs four different prompts once each (a story, Python code, a history explanation, French grammar), starting from a cold cache, instead of repeating the story. The expert cache carries over from one generation to the next, so a repeated prompt replays the same routing on a warm cache (91% hits instead of 89% for the story), and the story is among the fastest of the four: 10.7 tok/s, against 9.4–9.5 for the history and French answers, whose hit rate is 87%. Other apps closed (an open browser costs ~0.7 tok/s on this Mac). Strata's numbers are its published ones, not rerun here. Its Mac guide describes them as "medians of 5 interleaved runs" (story and short-question decodes of 200 tokens, a 4K-token prompt, no MTP) but does not give the prompts, the warmup, or whether a run repeats a prompt on a warm cache, so the comparison stays approximate.

Perplexity on 1,024 tokens of Transformers documentation with the same 3-bit experts; 8-bit dense layers pick the same top-1 token as BF16 at 95% of positions, 4-bit at 82%. 8 I/O threads. † a 4K prefill with a 12 GiB cache needs more free memory than this Mac had with its usual apps open (it ran at 60–70 tok/s on a quieter day); with 10 GiB it fits. Long prompts are prefilled in 1,024-token chunks, as Strata does on Macs under 96 GB. * measured before the read-pipeline and decode-kernel work, two runs of the same prompt (warm cache for the second). Strata's M4 runs use no MTP. Through the decode path itself (teacher-forced, one token per step, 512 tokens), perplexity is 3.3558 with BF16 dense layers and 3.3629 with 8-bit ones (+0.2%, same top-1 token at 94.5% of positions; data, script). ‡ Strata's quality table, as quoted by ModelFit: perplexity relative to its own Q8_0 build, on text we do not know, so it is not comparable with our absolute numbers (ours compare dense-layer precisions with the same 3-bit experts).

Decode after a 6K-token document

Strata's third Mac row: 200 tokens decoded after reading a long document, so the attention layers see ~6K earlier tokens and the prefill has just churned the expert cache.

The document is the 4K prompt's synthetic harbour records, extended to 6K tokens; ask: summarize them. The 6K prefill peaks at 25.4 GiB of device memory with a 12 GiB cache (24.0 with 10 GiB), so it needs other apps closed. Decode drops from 10.0 to 7.3: the cache holds what the prefill touched (56% of lookups missed over the whole run; 292 GB read from the SSD) and full attention covers 6K tokens. Still behind Strata's IQ3_XXS (7.7), on a faster Mac. The two cache sizes give the same answer for ~150 tokens, then different wording (the prefill groups experts differently, which changes the summation order). Reports: 12 GiB, 10 GiB.

A fair comparison

Strata publishes several expert quantizations; the one to compare with is the one closest in quality to ours, not the fastest:

• Quality: our experts are 3-bit MLX affine (group 64), between the two. We have not measured them against 8-bit experts (the FP8 source was deleted after conversion), so IQ3_XXS is the like-for-like row, and Q2_0 trades ~10% perplexity for its 13.5 tok/s.
• Bytes per expert: with a 16-bit scale and bias per 64 weights, ours take ~3.5 bits per weight, ~14% more than IQ3_XXS: more to read per miss and fewer experts per GiB of cache.
• Hardware: our M5 has ~28% more memory bandwidth than Strata's base M4 (153 vs 120 GB/s). Dense-layer decode time scales with it, so 10.0 tok/s on the M5 against 9.2 on the M4 does not show Transformers to be faster; with Strata on an M5 we would likely be behind. Only a run of both on the same Mac settles it.
• Protocol: Strata's prompts and warmup are not published. Ours: four different prompts, cold cache first.
• Speculative decoding: Strata leaves MTP off on 32 GB Macs ("MTP was slower"): with streaming, draft tokens touch more experts. On a 128 GB M5 Max it takes Q2_0 from 87.3 to 108.2 tok/s.

The two Macs

Apple's launch specs. Strata's guide says "M4 Mac mini 32 GB", which is the base M4 (the M4 Pro Mac mini comes with 24, 48 or 64 GB). The M5 has ~28% more memory bandwidth and its SSD reads 7.1 GB/s even for the 2.15 MB random reads experts need, so numbers measured on the two Macs are not directly comparable (see "A fair comparison").

Architecture

Model: Qwen3.8-Flash-Next, 48 decoder layers, hidden size 2,560, vocabulary 248,320. Every layer has a token mixer and an MoE block, each wrapped in a 4-stream hyper-connection. Token mixers: 36 Gated DeltaNet (linear attention) layers and 12 full-attention layers (one every 4), whose attention is sparse: an indexer keeps at most 2,048 earlier tokens per query. MoE: 512 routed experts (top-10, intermediate size 640) plus one shared expert per layer. Layer 1 adds a per-layer n-gram embedding (320M rows × 160). MTP and vision are not loaded.

Where every weight lives

Converted once from Qwen/Qwen3.8-Flash-Next-FP8 by convert_fp8.py (dense to BF16, experts to 3-bit, n-gram table kept FP8); MetalConfig quantizes the dense layers at load time.

Kernels, per decode step

Expert streaming

1. Loading: from_pretrained(..., experts_offload={...}) builds the model without expert weights and loads everything else normally; quantization_config=MetalConfig(bits=8) quantizes the dense layers.
2. Cache: one preallocated device pool of expert slots shared by all layers (LRU), with a per-layer table expert → slot kept on the host. 12 GiB hold ~6,000 of the 24,576 experts.
3. Each MoE layer: the router's choices are copied to the host (the one host synchronization per layer), misses are read on 8 I/O threads into recycled pinned host buffers (one contiguous read per expert, through descriptors opened once), copied to free slots without blocking, and the layer sends its slot indices in one small copy and runs gather_qmm on the pool.
4. Prediction: the routers of the next two layers run on the current hidden states; their guessed experts are read in the background, and the next layer's guesses are installed in the pool while the current layer computes.
5. Prefill: a 4K prompt touches nearly every expert of every layer; they go through the sorted gather_qmm 64 at a time so the transient bank stays small.

Decode speedups since the first published numbers

Each measured as alternating A/B pairs (4 prompts per run, 12 GiB cache), commits on the POC branch:

The MPS commit issue. PyTorch's MPS stream encodes small kernels into a command buffer and commits it only at the next synchronization (or under memory pressure). Decode is made of small kernels and synchronizes once per MoE layer, so after each routing wait the GPU idled while the host encoded the experts and the next layer, then ran them only at the next wait. Launching 1,000 small matmuls and then doing 106 ms of host work takes 213 ms; recording an event after the launches (it commits without waiting) brings it to 109 ms (reproducer). With the experts replaced by zeros, the same fix takes decode from 67 to 56 ms per token. It would be best fixed in PyTorch; Transformers works around it with experts_offload={"mps_commit": True} (the default on MPS).

Do these changes help other MoEs?

Two of the three decode changes are generic, one targets a family of models, one is Qwen3.8-only:

OLMoE-1B-7B (16 layers × 64 experts, 4-bit experts, 3.4 GB), same four prompts, 200 tokens, greedy, cold cache first; before is the branch just before the three changes (script, reports):

Streaming OLMoE gains ~3%, within run-to-run noise: its dense layers are small, so the GPU had little host encoding to wait for. With a cache that holds every expert, the gain comes from the first two answers, while the cache fills (66–67 against 47–48 tok/s); once every expert is resident and no layer synchronizes, both run at the same speed (79 / 84 against 72 / 85). Unrelated to these changes: with every expert resident, prefill takes a different matmul path (perplexity 8.2144 instead of 8.2953) and one of the four answers differs from the streamed one, before and after alike.

What still limits decode

The dense layers, measured by ablation: experts replaced by zeros, then each module category replays its first output instead of computing, which shows what it would save if it were free (data, script). Floor: reading the category's 8-bit weights once at the M5's 153 GB/s.

• Expert misses (10–16 ms per token before these changes): ~0.6 experts per layer and token are still being read when their layer runs; each wait is a whole read (~0.34 ms), so the guess missed them rather than being late. Most are first uses of an expert in the answer, which no cache policy avoids (see below): only better prediction of the next layers' experts or smaller experts help. Splitting reads, guessing wider (16 instead of 10 experts) and LFU did not.
• One host synchronization per MoE layer (48 per token): an MPS round trip costs ~210–250 µs even for a tiny op, whatever the mechanism (.cpu(), an event or torch.mps.synchronize).
• Rewriting the cache bookkeeping (pure Python trims) gained nothing measurable once the shared expert overlapped it.
• The Gated DeltaNet layers are at the memory-bandwidth limit: per layer, the merged 8-bit in-projection (2,560 → 16,480) takes 372 µs for 44.8 MB and out_proj 142 µs for 16.7 MB, ~120 GB/s, about what the M5 sustains; the two Metal kernels add ~70 µs (data). Only fewer bytes would help: 6-bit dense layers would read 25% less, but MetalConfig supports 2, 4 and 8 bits only, and 4-bit costs 9.3% perplexity.
• Left at 8-bit: the hyper-connections (~10 ms per token, ~5 ms floor; fusing the up-projection into the mixing kernel could save a few ms) and full attention (6.8 ms, ~5 ms floor).

Would a warmup or a smarter cache policy help? No

The routing of the four decode prompts (plus the story again) was logged and replayed through a cache simulator at the 12 GiB size (5,991 expert slots), counting only the experts the model asks for (no next-layer guesses, hence a few points below the measured hit rates). Belady's policy evicts the expert needed furthest in the future: no policy can beat it.

One 200-token answer touches 10.7–12.6k distinct experts, about twice what the cache holds, and 10–13% of its lookups are first uses. For the story, the optimum at 12 GiB equals that first-use floor (89.3%), and LRU reaches it with a 1.5× cache. Consecutive prompts share only 51–72% of their experts, so warming up gains about one point, and pinning "popular" experts loses up to 10. New experts keep appearing through the answer (1.7 per layer and token at the start, 0.2 by token 175), so long conversations would gain a little from a better policy, but a 200-token benchmark would not. Replay output, simulator, routing trace.

CUDA (preview): one A10G

The same converted model also runs on an NVIDIA A10G (24 GB) through the same package (run, reports): BF16 dense layers, 3-bit experts in an 8 GiB device cache, run by a Triton kernel that unpacks the MLX format in registers (matches PyTorch at bfloat16 precision for 2-8 bits, check). The pod has 83.5 GiB of RAM and 22.8 CPUs; the experts sit on a network file system.

Quality: decode-path perplexity (teacher-forced, 512 tokens) is 3.4366 on CUDA (BF16 dense) vs 3.3558 on the Mac (BF16 dense); the two agree on the top-1 token 95.3% of the time. The host tier (host_cache_bytes, host_preload) is now in the PR. An answer that differed in an earlier run came from CUDA decode not being bit-deterministic run to run, not from the host tier.

Early numbers. Decode is held back by the dense layers, which run as plain PyTorch ops on CUDA: the Metal kernels (Gated DeltaNet step, hyper-connections) have no CUDA counterparts yet, and the per-expert prefill path makes a 4K prompt slow (17 tok/s).

Known issues

• On the M5, the NAX kernels of mlx-quantization-metal-kernels v2 return wrong results for bfloat16/float16 inputs once a call has enough rows: sorted gather_qmm from 4 rows per expert (kernels-community#1332), and the dense quantized_matmul from ~30–128 rows depending on the shape (reproducer). float32 is correct, and so is every single-row (decode) call; prefill runs those calls in float32.
• Off Apple silicon and CUDA, quantized experts use a slow PyTorch reference. On CUDA, decode uses a Triton kernel; prefill still dequantizes expert by expert.

Files: transformers.patch (against huggingface/transformers 158f872eb) · raw reports, converter and benchmark.