Speculative Decoding and the Latency Ceiling for On-Device LLMs

For the last few years, I have worked on shipping large language models onto phones, the kind that runs in your pocket with no cloud round trip. And for most of that time, I kept running into the same uncomfortable question from teammates and app developers: we quantized the model, we pruned it, we compiled it down to almost nothing, so why does it still feel slow?

The honest answer is that the thing making it feel slow was never the size of the model. It was the way the model produces tokens or the words. And no amount of compression, however clever, can fix that on its own. This is a story about the actual ceiling that on-device models hit, why the usual toolkit cannot break through it, and what changed when my team stopped trying to shrink the model and started changing how it generates.

The bottleneck nobody...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE