Let's Build Our Own LLM (Part 1): Tokenization and Data Prep
A few years ago, people noticed that ChatGPT was strangely bad at counting the letters in a word, or at simple tasks like reversing a string. How could a model that writes working code fail at something a child can do?
The answer is buried in a step most people never think about. The model doesn't see letters. It doesn't even see words. Before a single neuron fires, your text gets chopped into chunks called tokens, and the model only ever sees those chunks. "Unbelievable" might arrive as three pieces: un, belie, vable. "Strawberry" might arrive as str, aw, berry which is exactly why the model struggled to count its r's. It never saw individual r's in the first place.
Tokenization is the invisible lens through which the model views all language. Get the lens wrong, and everything downstream is blurry.
This article is about that lens, and about...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
