Flash-Moe: Running a 397B Parameter Model on a Mac with 48GB RAM

Flash-Moe is a pure C/Metal inference engine that runs the 397B-parameter Qwen3.5 MoE model on a MacBook Pro with 48GB RAM at 4.4+ tokens/second by streaming 209GB of expert weights from SSD on demand. Key optimizations include FMA-optimized 4-bit dequantization kernels, deferred GPU compute for CPU/GPU overlap, Accelerate BLAS for linear attention layers, and relying on the OS page cache instead of custom caching. The system uses hand-written Metal shaders for all GPU operations and achieves production-quality output including tool calling at 4-bit quantization.