Running a 28.9M Parameter LLM on an $8 Microcontroller
When you think of large language models (LLMs), powerful GPUs with dozens or even hundreds of gigabytes of memory likely come to mind. That’s the kind of hardware it takes to run today’s cutting-edge models. It would be hard to imagine this type of algorithm running on a tiny microcontroller, yet one developer has managed to do exactly that. While certainly no frontier model, its 28.9 million parameters make it impressive to see running on an $8 development board all the same.
28 million parameters on an $8 chip
The project runs the language model entirely on an ESP32-S3, a microcontroller with just 512KB of SRAM, 8MB of PSRAM, and 16MB of flash storage available to it. Everything happens locally on the chip, with no cloud connectivity or external server involved. The generated text is written directly to a small display connected to the board at roughly 9.5 tokens per...
Copyright of this story solely belongs to hackster.io. To see the full text click HERE
