Article Summary (Model: gpt-5.5)
Subject: SSD-Streamed Gemma
The Gist:
TurboFieldfare is an Apache-licensed Swift + Metal inference runtime that runs the instruction-tuned Gemma 4 26B-A4B model on Apple Silicon Macs with about 2 GB of process memory. Instead of loading the full 14.3 GB installed model, it keeps shared weights and KV cache resident, then streams only the routed MoE experts needed for each token from SSD.
Key Claims/Facts:
- Bounded Memory: Keeps a ~1.35 GB shared core plus FP16 KV cache in memory; routed experts are fetched on demand.
- MoE Streaming: Uses router top-8 expert IDs, 16-slot per-layer LFU caches, and bounded parallel
preadcalls into Metal-visible buffers. - Benchmarks/Scope: Reports 5.1–6.3 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on a 24 GB M5 Pro; text-only Gemma 4 26B-A4B only, requiring macOS 26/Metal 4/Swift 6.2.
Discussion Summary (Model: gpt-5.5)
Consensus: Cautiously optimistic: commenters were impressed by practical local inference on small Macs, while emphasizing that performance depends heavily on SSD speed, OS page cache, and MoE-specific structure.
Top Critiques & Pushback:
mmapwas much slower on an 8 GB M2 because page faults are reactive, while explicitpreadcan fetch whole expert blocks and overlap I/O with GPU work (c49099165, c49099602, c49101455).Better Alternatives / Prior Art:
Expert Context:
pread, with ~41% expert reuse on the next token and ~57% within two tokens (c49103630).