Open Inference Networks: Democratic Access to Intelligence
Weights you cannot run are a file
An open model on your disk is only a large file until something runs it. If the only place that can run it is a rented data center, you are still a tenant.
I come at this from edge work. Models on a drone have to run on the device, with no network at all. That makes me sympathetic to the local-first argument, and also impatient with claims that ignore physics. I wrote about that shift in my piece on edge AI.
There are three ways to run a model outside a hyperscaler. I will take them in order of how well they work.
Tier one: run it yourself
This is the part of the story that has clearly arrived.
Tools such as llama.cpp and Ollama made local inference a one-line install. Quantization, which stores each weight in about 4 bits, cut...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
