A 30B Model Crawled on My Mini PC, But the Weights Were Innocent
There is a comfortable assumption in the local LLM world: if a model only activates a few billion parameters per token, it will run fast on modest hardware. Sparse mixture-of-experts models are sold on exactly that promise. GLM-4.7-Flash is 30 billion parameters on paper and lights up roughly 3 billion per token, so on a small machine, that ratio is the entire reason to bother with it. When it crawled on my mini PC, I assumed I had misconfigured something. I had not. The model was fighting the runtime, and for a while, the runtime was winning.
A quick disclosure, because it explains why I chased this so hard. I built ModelDirectory, and a fair amount of its unglamorous backend work runs on local hardware rather than a rented GPU. So, "can this model actually earn its place on a small box" is not a benchmark hobby for me....
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
