I Built a Python Compressor Using OpenAI's Tokenizer. Here's What Happened

LLMs spent billions of dollars learning the best subword dictionary that has ever existed. Somewhere around 200,000 pieces of language, ranked by how much they compress. And I kept looking at that dictionary and thinking: nobody is allowed to just use that for compression?

TokPress is a pure-Python lossless compressor that tokenizes input with OpenAI's o200k_base vocabulary, applies token-level LZ77, and entropy-codes the result with rANS. For many small, schema-homogeneous records (logs, JSON, telemetry) a trained dictionary takes the ratio to 0.2565, and a batch mode reaches 0.0875x — while compressing each record alone can inflate it past 1.0. It beats gzip on prose and loses honestly to zstd's trained dictionary.

So I built one. Pure Python, no native code, one weekend, way too many cups of coffee. It's called TokPress, it compresses by tokenizing with o200k_base (the tokenizer OpenAI's models use), applying token-level LZ77, and entropy-coding the result...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE