LLM Token Counter
Estimate how many tokens your text will use — before you send it to a model or pay for it. This runs a BPE-aware approximation entirely in your browser: the real GPT pre-tokenisation step, plus a byte-calibrated subword model. No text is uploaded.
What a token actually is
Language models don't read characters or words — they read tokens, the units their tokeniser carves text into. A token is often a whole common word, sometimes a word fragment, sometimes a single punctuation mark or space. The rough rule of thumb is about 4 characters per token for English, or three tokens for every four words — but that average hides a lot of variation, which is exactly what this tool tries to capture.
How byte-pair encoding works
Modern tokenisers use byte-pair encoding (BPE). Training starts with raw bytes and repeatedly merges the most frequent adjacent pair into a new symbol, thousands of times over. The result is a vocabulary where common sequences ("the", " and", "ing") become single tokens, while rare strings stay split into smaller pieces. It's why "hello" is one token but a random string like "xqzkw" might be four.
Why this is an approximation
An exact count requires the model's full merge table — tens of thousands of learned rules, often a megabyte or more of data. Rather than ship that, this tool reproduces the part of the pipeline that does most of the work and needs no table: the pre-tokenisation regex. Real GPT tokenisers first split text with a pattern that isolates words (with their leading space), runs of punctuation, whitespace, and — importantly — numbers in groups of at most three digits. Each resulting piece is then estimated by its UTF-8 byte length at roughly four bytes per token.
This captures the behaviours a naive "characters ÷ 4" misses entirely: numbers cost more than their length suggests, code and punctuation are token-dense, and non-Latin scripts like Chinese, Japanese, or Korean use several tokens per character because each character is multiple bytes. For ordinary English prose the estimate lands within a token or two of the real count; for code and mixed content it stays far closer than the simple rule.
When you need the exact number
For billing-critical or limit-critical work, confirm with the official tokeniser — OpenAI's tiktoken, or the count returned in an API response's usage field. Treat the number here as a fast, private, dependable estimate for drafting prompts, checking you're under a context window, and ballparking cost.
Privacy
Everything is computed locally in your browser. Your text is never transmitted, stored, or logged — paste a confidential prompt or document without concern.