The hardware specifications read like a misprint. A $599 Mac mini. 16 GB of RAM. A 122-billion-parameter language model running on it, producing coherent output, with memory usage flatlined at 9 GB.
That doesn’t add up. And the reason it doesn’t add up is exactly why this matters.
A model this size has no business fitting on that machine. In standard 16-bit format, it weighs roughly 240 GB. Even aggressively compressed to 3-bit, it sits at 54 GB on disk, more than three times the machine’s entire RAM. By every conventional rule of inference, it should crash on load, not generate tokens.
It runs. And the lesson it teaches along the way rewrites something fundamental about how we think about local AI.
The Problem with the Memory Wall
For the past two years, the story of local AI has been a story about quantisation, squeezing model weights into fewer bits so they fit on consumer hardware. It worked. Models that once required a data centre can now run on a MacBook. But there’s a ceiling, and most people assumed they’d hit it around 30–70 billion parameters on consumer machines. Manjunath Janardhan’s experiment, published in May 2026 in Data Science Collective, challenges that ceiling directly.
Using Qwen3.5-122B-A10B, Alibaba’s 122-billion-parameter Mixture-of-Experts model (MoE) expert streaming implemented through TurboQuant-MLX, he ran a frontier-class language model on the cheapest Mac Apple sells.
The key insight is that for a sparse MoE model, the memory wall is really a disk-bandwidth wall. And those two problems have very different solutions.
How a 122B Model Fits in 9 GB of RAM
What Makes MoE Different
Most large language models are dense: every token processed passes through every parameter in the network. A 7B dense model uses all 7 billion weights for every single token it generates. Memory requirement scales directly with model size.
Mixture-of-Experts models work differently. The Qwen3.5-122B-A10B has 256 expert sub-networks, but each token only activates roughly 8 of them. The “A10B” in the model name means approximately 10 billion parameters are active per forward pass, out of 122 billion total. The rest sit dormant for that token.
That’s the key. You don’t need 122B parameters in RAM. You only ever need the 8 active experts for any given token.
Expert Streaming: The Mechanism
The insight Janardhan implements is called expert streaming, or per-expert disk streaming. Instead of loading all 256 expert weights into RAM before inference begins, the system pages only the active experts for each token directly from SSD, uses them, and discards them. A small cache retains recently used experts to avoid redundant reads on common patterns.
The math becomes striking quickly. 256 full experts stored on disk: 54 GB. 8 active experts loaded per token: a fraction of that. The resident memory footprint peaks at roughly 9 GB, less than a typical browser tab count on a developer’s machine.
Apple’s unified memory architecture is what makes this viable on Mac hardware. On a conventional machine, CPU and GPU have separate memory pools; shuttling data between them creates latency. On Apple Silicon, unified memory means the GPU can stream directly from NVMe SSD via the CPU cache, with no copy overhead. The SSD becomes an extension of the memory hierarchy, not a separate storage tier.
TurboQuant-MLX: The Compression Layer
Expert streaming handles the where and which experts to load. TurboQuant-MLX handles how small compressing each expert before it even hits the disk.
TurboQuant is an algorithm from Google Research (Zandieh et al., presented at ICLR 2026) that applies random QR rotation followed by Lloyd-Max scalar quantisation to compress KV cache and weight tensors down to 3-bit precision with near-zero quality loss. The rotation step is what prevents the accuracy collapse that normally destroys models below 4-bit: it decorrelates the values before quantisation, eliminating the outlier spikes that standard affine methods can’t handle.
Community implementations on Apple Silicon confirmed 5× compression on models like Gemma 4 26B and Qwen3 9B with stable perplexity across all tested prompt lengths. Applied to the 122B MoE, the combined effect of quantisation plus streaming is what turns a 240 GB model into a 9 GB resident footprint.
What This Looks Like in Practice
Janardhan’s methodology is worth noting for its deliberateness. He didn’t start at 122B. He started with a 35B model to validate the streaming approach, watched it fit under 4 GB of RAM, and then scaled the same code to the 122B to find the limits. The 35B fit. The 122B fit. Both produced coherent, bit-identical output with no swap activity and no system tweaks.
In terms of speed, expert streaming has a real cost: SSD throughput becomes the bottleneck, not GPU compute. Tokens per second drop compared to a fully RAM-resident model. But for developers who need frontier-scale reasoning for code review, document analysis, long-context tasks, or local agentic workflows, and who cannot justify cloud API costs at scale, a slower frontier model that runs free and locally is frequently the right trade.
The ecosystem building around this approach is moving quickly. SwiftLM (from SharpAI) implements a rewritten SSD expert streaming pipeline claimed to achieve a 10× speedup over naive streaming, with dual-model speculative decoding support. Rapid-MLX supports the Qwen3.5-122B directly via a one-line serve command. Multiple community forks are converging on production-quality implementations. Python 3.11+ on any Apple Silicon Mac is sufficient to run it today.
What This Actually Means
One observation from Manjunath Janardhan’s original analysis deserves special attention: “for a sparse MoE, the memory wall is really a disk-bandwidth wall.“ That insight fundamentally changes how we think about local AI constraints.
For years, the assumption was simple: large language models require large amounts of RAM, making cloud deployment the only practical option. Expert streaming challenges that belief. Rather than loading an entire model into memory, Mixture-of-Experts (MoE) architectures activate only the experts needed for a given token, shifting the primary bottleneck from RAM capacity to SSD bandwidth.
This matters because modern NVMe storage is fast, affordable, and increasingly abundant. As a result, 100B+ parameter models may become accessible on consumer hardware not as experimental projects, but as practical tools for everyday development.
The implications extend beyond performance. Every token processed through local inference avoids API costs, data transfer, and external logging, making frontier-scale AI more attractive for developers, researchers, and organisations working with sensitive data.
The ecosystem is already moving quickly. Google’s TurboQuant research, combined with rapid adoption across MLX, PyTorch, and llama.cpp communities, suggests that this is more than an isolated experiment. It represents a broader shift in how frontier-scale models are deployed, optimised, and ultimately brought closer to the edge.
Key Takeaways
- Qwen3.5-122B-A10B is a 256-expert MoE model that activates only 8 experts per token, enabling most of its parameters to stay off-chip at any given moment.
- MoE expert streaming pages only the active experts from SSD per token, reducing the 54 GB on-disk model to a ~9 GB RAM footprint with no swap.
- TurboQuant-MLX applies random QR rotation and Lloyd-Max quantisation for 3-bit compression with near-zero accuracy loss.
- Apple’s unified memory architecture is the enabling hardware: GPU can stream directly from NVMe without copy overhead, making SSD the effective memory extension.
- The memory wall for sparse MoE models is really a disk-bandwidth wall, a fundamentally different constraint with fundamentally different solutions.
- The ecosystem is moving fast: SwiftLM, Rapid-MLX, and multiple community forks already support the 122B model with one-line serve commands on Apple Silicon.
- Trade-offs are real: expert streaming reduces tokens-per-second compared to fully RAM-resident inference. Frontier reasoning at a lower speed versus no frontier reasoning at all.
- Python 3.11+ on any Apple Silicon Mac is sufficient to begin experimenting today.
- Privacy and cost implications are significant: local frontier inference eliminates API logging, billing, and data-egress concerns for sensitive or high-volume workloads.
Conclusion
There’s something quietly disorienting about the $599 Mac mini running a 122B-parameter model. Not because the engineering is magic, it isn’t. Every technique involved is documented, reproducible, and available in open-source repositories. What’s disorienting is that the constraint everyone accepted as fundamental turned out to be, in part, a category error.
We said: This model won’t fit in RAM. That was true. We concluded: this model can’t run on consumer hardware. That conclusion assumed the model needed to fit in RAM all at once. For a sparse MoE, it doesn’t.
The question the experiment implicitly asks is what other constraints we’ve been accepting on similar grounds, and what it would take to look at them from a different angle. The hardware industry is clearly paying attention. The software ecosystem around Apple Silicon inference is among the fastest-moving in open-source AI right now.
If 122B parameters fits on a $599 machine today, where does that leave the assumptions we’re making about 2027?
