Tokenizer Visualizer
Type anything and watch it shatter into the colored chunks a real LLM actually sees — then understand why models can't count the r's in strawberry.
Runs 100% in your browser — check the network tabHover — or tap on mobile — any chip to see the exact token string, its ID, and its byte length.
Under the Hood
Byte Pair Encoding starts from raw bytes — 256 possible values, one per byte. It scans a huge pile of text and repeatedly merges whichever adjacent pair of tokens shows up most often, turning that pair into one new, single token. Do that on the order of 100,000 times and you end up with a vocabulary where common words are one token each, and rare or unusual words get spelled out as several smaller pieces instead.
This page uses cl100k_base, the encoding behind GPT-3.5 and GPT-4. Every model family builds its own vocabulary the same way, so exact token counts differ from model to model — the mechanism you’re watching here is universal, the specific numbers are not.
Why this matters beyond trivia
- Spelling — rare words get chopped into subword pieces, so counting letters means reconstructing spelling from fragments the model never sees as individual characters.
- Math— digit grouping doesn’t follow place value, especially once separators are added, which is part of why arithmetic on large numbers is unreliable.
- Pricing — verbose formatting like indentation or repeated punctuation burns tokens without adding meaning.
- Multilingual fairness — languages under-represented in training data cost more tokens, and therefore more money and less usable context, for the exact same meaning.
Go deeper
cl100k_basefirst splits text with a regex — separating letters, digits, punctuation, and whitespace runs into their own chunks — before running BPE merges within each chunk. That regex pre-split, not just the merges, is part of why numbers group in short runs and why casing changes results so much: “HELLO” and “hello” are different starting chunks before any merging even happens.
Bytes that BPE hasn’t learned to combine yet stay as single-byte tokens. That’s what the Bytes view exposes for multi-byte characters: a token that represents one raw byte of a 3-byte Chinese character, or one byte of a 4-byte emoji, isn’t a valid character on its own — decode it by itself and you get nothing printable, which is exactly why this demo shows raw bytes instead of pretending every token is a clean character.