Breaking Down Text
Tokenizers split text into smaller pieces called tokens. Common words are often single tokens, while rare words get split into subwords. "Understanding" might become ["under", "standing"].
Count tokens in your text for ChatGPT, Claude, Gemini, and other AI models. Estimate your API costs before making requests.
Note: Uses o200k_base encoding (GPT-4o / GPT-5). Other providers use different tokenizers, so counts may vary slightly.
Related: For guidance on how document size maps to model context windows, see our LLM Context Window Comparison tool.
Based on token count at current model prices (input only):
* Input token cost only. Use our LLM Calculator for detailed pricing.
Tokenizers split text into smaller pieces called tokens. Common words are often single tokens, while rare words get split into subwords. "Understanding" might become ["under", "standing"].
English is the most token-efficient. Other languages, especially those with non-Latin scripts (Chinese, Arabic, Hindi), typically use 2-3x more tokens for the same content.
Code often uses more tokens due to special characters and formatting. Spaces, newlines, and punctuation all count as tokens. Minimize whitespace to save tokens.
Users upgrading from legacy GPT-4 Turbo to the new GPT-5.2 or GPT-5 mini
will see an immediate reduction in token usage. The GPT-5 architecture standardizes on the
o200k_base tokenizer, which is significantly more efficient at encoding code (Python, JavaScript)
than the older cl100k_base. Benchmarks show that code-heavy prompts consume 15-20% fewer
tokens on GPT-5, effectively lowering API costs prior to any logic optimization.
Modern reasoning models like GPT-5.2 and DeepSeek V3.2 generate substantial "Chain of Thought" (CoT) tokens during processing. You are billed for these tokens even though they are discarded before the final response. For these models, visible output typically represents only 20-30% of total billed tokens. We recommend buffering your budget by 4x when using reasoning-heavy endpoints.
While Gemini 3 Flash has improved non-English efficiency, most tokenizers remain biased toward English. RAG pipelines for languages like Hindi, Arabic, or Chinese using Gemini 3 Flash will incur significantly higher costs, often 2x to 3x per semantic unit compared to English. Use this counter to audit multilingual prompts specifically for non-Latin scripts before deployment.
See how different text types tokenize
Maximum tokens (input + output) for popular models
| Model | Context Window | ≈ Words |
|---|---|---|
| DeepSeek V3.2 | 128,000 tokens | ~96,000 words |
| GPT-5.2 | 400,000 tokens | ~300,000 words |
| Claude Opus 4.5 | 200,000 tokens | ~150,000 words |
| Claude Sonnet 4 | 1,000,000 tokens | ~750,000 words |
| Gemini 3 Pro | 1,000,000 tokens | ~750,000 words |
| Gemini 3 Flash | 1,000,000 tokens | ~750,000 words |
A token is a chunk of text that AI models process. In English, a token is roughly 4 characters or about 0.75 words. For example, the word 'hamburger' is split into 'hamb', 'urger' - 2 tokens Common words like "the" or "is" are typically 1 token each.
This tool uses the official OpenAI tokenizer (o200k_base), providing 100% accurate counts for GPT-4o, GPT-5, and other OpenAI models. Claude, Gemini, and other providers use different tokenizers, so counts may vary by approximately 5%.
Each AI provider develops their own tokenizer optimized for their models. OpenAI's GPT models use tiktoken, Anthropic's Claude uses a different tokenizer, and Google's models have their own. The differences are usually small but can affect pricing calculations.
To reduce tokens: (1) Be concise - remove unnecessary words, (2) Remove extra whitespace and newlines, (3) Use shorter variable names in code, (4) Avoid repeating context, (5) Use abbreviations where appropriate. Every token saved reduces your API costs.
Yes, spaces and punctuation are often included in tokens. A space before a word is typically merged with that word into a single token. Punctuation marks like periods, commas, and quotes are usually separate tokens.
In English, 1000 words is approximately 1,333 tokens (using the ~0.75 words per token rule). However, this varies based on word complexity. Technical text with specialized terms may use more tokens, while simple prose uses fewer.