The model doesn't fit: the levers to run it anyway

A five-minute community talk. A 400-billion-parameter Mixture of Experts weighs about 200 GB even at 4 bits: it fits neither on a 128 GB MacBook nor on an H100. And yet in recent months people have run 744-billion-parameter models on a laptop with 25 GB of RAM. The projects change every week, the levers stay the same, and each one has a price: quantization cuts bytes but below 4 bits breaks JSON and tool calling; experts stay on SSD and move into RAM only when the router picks them, at the cost of latency; loading them ahead hides the wait, but you pay when the prediction is wrong.

Gallery