What Are Tokens in AI? A Simple Guide With Real Examples (2026)

Short answer: tokens are the small pieces of text that AI models such as ChatGPT, Claude and Gemini read and write. A token can be a whole word, part of a word, a number, a space or a punctuation mark. In English, one token is roughly four characters, or about three quarters of a word. Tokens matter because AI companies count them to set limits and to bill you: you pay for the tokens you send in and the tokens the model writes back.

So what are tokens in AI, exactly? Every time you type a question into an AI chatbot, the model never sees your words the way you do. It first cuts the text into tokens, turns each token into a number, and works with those numbers. The answer comes back the same way, one token at a time.

This guide explains what AI tokens are, how text is split into them, how many words a token is, why other languages use more tokens, and what tokens cost. The examples come from running real text through OpenAI’s open-source tokenizer, and every price and limit was checked against the official documentation of OpenAI, Anthropic and Google in November 2026.

Key takeaways

• A token is a chunk of text, usually a word or part of a word, that an AI model treats as one unit.
• In English, 1 token is about 4 characters, and 100 tokens are about 75 words (OpenAI’s rule of thumb).
• Tokens are not the same across companies. The same sentence can produce a different number of tokens on ChatGPT, Claude and Gemini.
• Other languages often need more tokens than English for the same meaning, which makes them cost more.
• You pay for input and output tokens, and output tokens usually cost several times more than input tokens.
• Hidden “reasoning” tokens count as output, so a short answer can still use a lot of tokens.
• The context window is how much text, counted in tokens, a model can take in a single request. For today’s leading models it is often around 1 million tokens.

What is a token in AI?

A token is the basic unit of text that a large language model (LLM) reads and produces. Before a model can do anything with your message, a piece of software called a tokenizer splits the text into tokens and swaps each one for a number from a fixed list. That list is called the model’s vocabulary, and for OpenAI’s recent tokenizers it holds roughly 100,000 to 200,000 entries.

Common words such as “the”, “cat” or “model” are usually a single token. Longer or rarer words are split into several pieces. Spaces, capital letters and punctuation all affect the split, so a token is not the same as a word, a syllable or a letter.

A simple way to picture it: think of tokens as the Lego bricks of language for an AI model. The model has a box of bricks of different sizes. It builds every sentence it reads and writes from those bricks, picking the biggest brick that fits wherever it can.

How tokenization works

Most modern models use a method called byte-pair encoding (BPE) or a close relative. The idea is simple, even if the details are technical:

1. Start small. Training begins with individual characters, or the raw bytes that make up text.
2. Merge common pairs. The pieces that appear next to each other most often in a huge amount of text are joined into a new, bigger piece.
3. Repeat. This keeps going until the vocabulary reaches its target size. Frequent words end up as one token; rare words stay split into smaller parts.
4. Use the same rules every time. When you send a message, the tokenizer applies those learned merges to cut your text into tokens, then turns them into numbers for the model.

Because the vocabulary is learned from training data, a tokenizer is efficient at text that looked like its training data, such as everyday English, and less efficient at unusual text such as long web addresses, rare names or languages it saw less of.

Real examples: how text is split into tokens

To show what this looks like in practice, we ran some text through o200k_base, the open-source tokenizer that OpenAI’s tiktoken library uses for GPT-4o and the GPT-5 family of models. The vertical bars show where one token ends and the next begins.

A few things stand out:

• Simple English is efficient. Both full sentences used about one token per word, with the full stop as its own token.
• Spaces travel with the word. Most tokens include the space in front of them, so ” sat” with a space is a different token from “sat” without one. Capital letters also change the token.
• Numbers and dates split oddly. The year 2026 became two tokens, “202” and “6”. This is one reason AI models can stumble on arithmetic with long numbers.
• Web addresses are expensive. A 41-character URL needed 16 tokens, about one token for every 2.6 characters, because brand names and slugs are rare in training data.
• Models do not see letters. “strawberry” arrives as three chunks, not ten letters. That is a commonly cited reason why chatbots have historically struggled to count the letters in a word.

How many words is a token?

There is no single answer, because every company uses its own tokenizer. These are the rules of thumb the AI companies publish themselves:

The Claude rows show why these ratios are only estimates. When Anthropic introduced a new tokenizer with Claude Opus 4.7, it warned that identical text can now come out anywhere from the same number of tokens to about 35% more, depending on what kind of text it is. The price per token did not change, so some workloads became more expensive simply because the text was cut into more pieces.

For quick planning in English, assume about 1.3 to 1.8 tokens per word depending on the model. For anything that affects a budget, count the real tokens with the company’s own tool.

Why other languages use more tokens

Tokenizers are trained on text that is heavily weighted towards English, so they learn bigger, more efficient chunks for English than for many other languages. We ran the same simple question, “Where is the nearest train station?”, through OpenAI’s current tokenizer (o200k_base) and its older one (cl100k_base, used by GPT-4 and GPT-3.5):

This is one short sentence, not a benchmark, but it shows two things. Newer tokenizers have narrowed the gap a lot: the Urdu sentence dropped from 33 tokens to 12. And the gap has not disappeared: Urdu and Hindi still need more than one and a half times as many tokens as English for the same question.

Research backs this up at scale. A NeurIPS 2023 study by researchers at the University of Oxford found that when one passage is translated into different languages, its token count can vary by as much as 15 times. Because AI services charge per token, that means people writing in some languages pay more, wait longer and fit less text into the model for the same request.

Input, output, cached and reasoning tokens

When you use an AI model through an API, the bill is split into different kinds of tokens:

• Input tokens are everything you send: your question, any instructions, documents you paste in, earlier messages in the conversation, and even tool definitions.
• Output tokens are the text the model writes back. They are priced higher than input tokens, typically five times higher or more.
• Cached tokens are input tokens the provider has already processed recently, such as a long system prompt you reuse. They are billed at a small fraction of the normal input price. On Claude Sonnet 5.5 and GPT-6.1 Sol, for example, a cache read costs 5% of the standard input price.
• Reasoning tokens (also called thinking tokens) are produced when a model “thinks” before it answers. You usually do not see them, but OpenAI confirms you pay for them at the output rate, and Google’s Gemini prices output “including thinking tokens”.

Reasoning tokens explain a common surprise: a model can give you a three-line answer and still bill you for thousands of output tokens, because most of the work happened in its hidden thinking.

Do images, audio and video use tokens?

Yes. Modern models are multimodal, and everything you send is converted into tokens, not just text:

• Images: Anthropic says Claude charges one visual token for each 28 × 28 pixel patch of an image, so bigger images cost more. Its documentation gives the example of an A4 page scanned at 130 DPI (1075 × 1520 pixels) costing 2,145 visual tokens.
• Audio: Google’s Gemini documentation estimates a 60-second audio clip at about 1,920 tokens (32 tokens per second).
• Video: the same documentation estimates a 60-second video at about 15,780 tokens (roughly 263 tokens per second).

So sending a screenshot or a short video can use as many tokens as several pages of text.

What is a context window?

The context window is the upper limit on how many tokens fit into a single request to a model. It has to hold everything at once: your instructions, the conversation so far, any documents, and the answer the model writes. If a conversation grows past the limit, older parts have to be dropped or summarised.

Context windows have grown fast. As of November 2026:

A million tokens is a lot of text: about 555,000 words of English on Anthropic’s current estimate. But a bigger window is not free: you pay for every input token in every request, and some providers charge higher rates once a prompt passes a size threshold. OpenAI, for example, doubles the input price for GPT-5.5 prompts over 272,000 tokens.

How much do AI tokens cost?

API prices are quoted per million tokens, with separate prices for input and output. Here are standard prices for a selection of current models, taken from each company’s official pricing page on 7 November 2026:

Prices change often, so always check the provider’s pricing page before you budget. All three companies also offer a batch option at half price for work that does not need an instant answer.

A worked example

Imagine a small business runs a support chatbot on Claude Sonnet 5.5. It handles 10,000 conversations a month, and each one uses about 1,500 input tokens and 500 output tokens.

• Input: 10,000 × 1,500 = 15 million tokens × $2 per million = $30
• Output: 10,000 × 500 = 5 million tokens × $10 per million = $50
• Total: about $80 a month, before any savings from caching or batch processing.

Notice that output is a quarter of the tokens but more than half of the cost. Keeping answers focused is often the quickest way to cut an AI bill.

One more catch: a lower price per token does not always mean a lower bill. Because tokenizers differ, the same text can produce more tokens on one model than another, and reasoning models may use extra hidden tokens. OpenAI’s own guidance is to compare the total cost per successful task, not the headline price. If you run AI workloads in the cloud, our guide to cloud cost optimisation covers the wider picture.

What do ChatGPT, Claude and Gemini subscriptions have to do with tokens?

If you use the chat apps on a monthly plan rather than the API, you do not pay per token. But tokens still sit behind the scenes. Long conversations, large file uploads and heavy reasoning all use more tokens, which is why plans have usage limits and why a very long chat can hit a limit sooner than several short ones.

What tokens mean for website owners

Tokens also affect how AI search tools read the web. When ChatGPT search, Perplexity or Google’s AI features fetch a page to answer a question, the text they take from it goes into a model’s context window alongside many other sources. A few practical points follow from that:

• Put the answer early. A clear, direct answer near the top of the page is easier for any reader, human or AI, to find and quote.
• Keep pages clean. Navigation, cookie banners and repeated boilerplate are all extra text a tool has to wade through before it reaches your content.
• Control access first. None of this matters if AI crawlers cannot reach your site. Our guide to AI crawlers explains which bots fetch pages for AI search, and our free AI Crawler Checker shows whether they can reach yours.
• Consider an llms.txt file. It gives AI tools a compact map of your best pages. Read what llms.txt is for an honest look at what it does and does not do.

How to count tokens

Rules of thumb are fine for rough planning, but each company gives you a way to count exactly:

• OpenAI: the online Tokenizer tool, or the open-source tiktoken library for code.
• Anthropic: the token counting endpoint in the Claude API, which returns the input token count for a message before you send it.
• Google: the countTokens method in the Gemini API.
• After the fact: every API response includes the input and output token counts that were actually billed.

For a quick check without writing code, paste your text into our free AI token calculator. It gives exact OpenAI counts, clearly labelled estimates for Claude and Gemini, and the cost per request and per month on 12 models. Your text stays in your browser.

Counting with the right tool matters. A count from OpenAI’s tokenizer is exact for OpenAI models but only an estimate for Claude or Gemini.

How to use fewer tokens

• Write shorter instructions. Cut repeated or unnecessary wording from prompts you use again and again.
• Ask for shorter answers. Set a length limit or ask for bullet points. Output tokens cost the most.
• Use prompt caching for long instructions or documents you send repeatedly.
• Use batch processing for anything that can wait, such as overnight reports. It is half price at all three providers.
• Pick the right model. Small, fast models cost a fraction of flagship models and handle simple tasks such as classification well.
• Trim conversation history. In long chats, summarise earlier turns instead of resending everything.
• Resize images before you send them, since bigger images cost more tokens.
• Set a lower reasoning effort on thinking models when the task is simple.

AI tokens vs crypto “AI tokens”

If you searched for “AI tokens” and found cryptocurrency pages, that is a different thing with the same name. Crypto AI tokens are digital coins linked to AI-themed blockchain projects. They have nothing to do with how language models read text. This guide is only about tokens in the language model sense.

Frequently asked questions

What is a token in AI in simple terms?

A token is a small chunk of text, often a word or part of a word, that an AI model reads as one unit. Models split everything you type into tokens and write their answers one token at a time.

How many tokens is 1,000 words?

In English, roughly 1,300 to 1,800 tokens, depending on the model. OpenAI’s rule of thumb of 100 tokens per 75 words gives about 1,333 tokens. On Anthropic’s current tokenizer, it is closer to 1,800.

Why do AI companies charge by the token?

Because the work a model does grows with the number of tokens it reads and writes. Counting tokens is a fair way to measure how much computing each request uses.

Are tokens the same on ChatGPT, Claude and Gemini?

No. Each company uses its own tokenizer, so the same text can produce a different number of tokens on each one. Always count tokens with the provider’s own tool.

Why are output tokens more expensive than input tokens?

The usual explanation is that a model can process your whole input in parallel, but it has to write its answer one token at a time, which uses more computing per token. Whatever the reason, output is priced higher at every major provider.

What happens if I go over the token limit?

Through an API, the request fails or the answer is cut off when it reaches the maximum output length. In chat apps, very long conversations may lose early details or hit a usage limit.

Do spaces and punctuation count as tokens?

Yes. Punctuation marks are often separate tokens, and spaces are usually attached to the word that follows them. Extra spaces, line breaks and symbols all add tokens.

Sources

• OpenAI Help Center: What are tokens and how to count them?
• OpenAI API: Pricing
• OpenAI API: GPT-5.5 model page
• OpenAI tiktoken (open-source tokenizer)
• Anthropic Claude Platform: Pricing
• Anthropic Claude Platform: Claude Sonnet 5.5 overview
• Anthropic Claude Platform: What’s new in Claude Opus 4.7 (tokenizer change)
• Anthropic Claude Platform: Image token cost
• Google Gemini API: Understand and count tokens
• Google Gemini API: Pricing
• Petrov et al. (NeurIPS 2023): Language Model Tokenizers Introduce Unfairness Between Languages