What Is a Token? The Unit Behind LLM Pricing, Limits, and Context Windows

Look at any generative AI pricing page and you will find rates quoted “per million tokens.” Model spec sheets do the same: “a context window of 200K tokens.” The token shows up everywhere — and it is the one unit you cannot avoid if you want to understand the cost and limits of using AI. This article explains the concept based on official documentation.

The short version: a token is the smallest unit in which a large language model (LLM) reads and writes text — and pricing, context windows, and rate limits are all counted in it.

What is a token? AI reads in fragments, not characters

According to Anthropic’s official glossary, tokens are “the smallest individual units of a language model, and can correspond to words, subwords, characters, or even bytes (in the case of Unicode)“¹. Microsoft’s documentation likewise explains that when you work with an LLM, text is first broken by a tokenizer into units that may be “words, character sets, or combinations of words and punctuation”².

The key point is that a token is neither a single character nor a whitespace-delimited word. As one Japanese-language explainer puts it, tokens are fragments — frequently occurring strings bundled together according to a dictionary defined by the model’s developer⁴. Where humans read text as characters and words, an AI processes it as a sequence of these fragments.

There are three broad tokenization approaches — word-level, character-level, and subword-level². OpenAI’s GPT models, for example, use a subword method known as Byte-Pair Encoding (BPE)². The set of tokens a model knows is called its vocabulary; each token is assigned an ID, and text is converted into a sequence of those IDs before processing².

Text generation also proceeds one token at a time. The model predicts the most probable next token, appends it to the sequence so far, and uses that as the input for the next prediction — building the final output iteratively². The way a chatbot’s answer trickles onto the screen from the beginning is this mechanism made visible.

Japanese consumes more tokens than English

The same content can produce very different token counts depending on the language. For Claude, one token represents approximately 3.5 English characters, though the exact number varies by language¹.

Japanese tends to be at a disadvantage here. Japanese-language explainers note that because tokenizer dictionaries are built mostly around frequent English patterns, Japanese text of the same content tends to break into more tokens than its English equivalent⁴. Exactly how much more depends on the model, the tokenizer, and the text itself — but the tendency that “saying the same thing in Japanese costs more tokens than in English” is worth keeping in mind when estimating cost and capacity.

Why it matters: cost, capacity, and speed are all metered in tokens

The practical reason to understand tokens is that every major number around LLMs is denominated in them.

First, pricing. Most generative AI services use token-based pricing: the cost of each request depends on the number of input and output tokens, and pricing might differ between input and output². Japanese-language explainers summarize the typical formula as input tokens times the input rate plus output tokens times the output rate⁴. Note also that in a chat, the prior conversation history is included in the input every turn, so the longer a conversation runs, the more input tokens each request consumes⁴. For tracking actual usage, a cost-management tool such as ccusage helps.

Second, how much text fits at once. LLMs have a maximum number of tokens for input and output, often expressed as a combined maximum context window covering both². If you have ever fed a long document to an AI and had it cut off partway, you hit this ceiling. We cover the details in our explainer on context windows.

Third, rate limits. Generative AI services enforce a maximum number of tokens per minute (TPM)². When embedding AI into a production system, TPM is often the factor that caps throughput.

You can count tokens before you send

There are also ways to know in advance how many tokens a prompt will be. Anthropic provides a token counting API that determines the number of tokens in a message before sending it to Claude, useful for proactively managing rate limits and costs or optimizing prompts to a specific length³. In the official documentation’s example, the system prompt “You are a scientist” combined with the message “Hello, Claude” counts as 14 input tokens³.

One caveat: the way tokens are counted — the tokenizer itself — can change between model generations. According to Anthropic’s documentation, Claude Fable 5 and Claude Mythos 5 use the tokenizer introduced with Claude Opus 4.7, which produces roughly 30% more tokens than earlier models for the same text³. The same passage of text does not have one universal token count — knowing this saves you from misestimating costs when migrating between models.

Tokens are a backstage unit you rarely need to think about day to day, but they are the common currency of all three constraints on LLM usage: cost, capacity, and speed. Whenever you need to optimize AI spending or judge the limits of long-document processing, this is the concept to reach for first.