$10 Microcontroller Runs a 135M-Parameter LLM Entirely Locally
As long as you have enough memory, it is possible to run even a frontier large language model (LLM) on just about any hardware. Big, fancy GPUs aren’t needed. A somewhat modern CPU will do the job. To be clear, it wouldn’t do the job well (it would be extremely slow) but it would run the model.
Microcontrollers are another story entirely. Memory is often measured in kilobytes on these chips, so running an LLM that is anything more than a toy would seem to be impossible. But YouTuber Dsn Industries’ latest project showed that this is not the case. He pushed an ESP32-S3 to its limit and demonstrated a 135-million-parameter language model running entirely locally on the processor.
The ESP32-S3 development board Dsn Industries used has 8 MB of PSRAM, which is quite a lot for a microcontroller, but nowhere near enough for a 135-million-parameter LLM. To make the...
Copyright of this story solely belongs to www.hackster.io. To see the full text click HERE
