Count tokens before designing an LLM feature

For a developer building an international assistant, a token budget should cover everything sent to the model and enough room for the answer. Tokens are the units a model’s tokenizer produces, not a fixed number of words or characters. A short-looking message can therefore consume more capacity than a longer-looking message in another language or format.

The tiktoken repository provides a tokenizer library for supported OpenAI encodings. Its examples show selecting an encoding and converting text to token identifiers. The exact tokenizer depends on the model. Treat a characters-per-token estimate as a rough planning aid, then count representative inputs with the intended encoding.

Allocate the request before filling it

Consider a hypothetical assistant with an application-imposed budget of 16,000 tokens. This is a teaching budget, not a claim about any model’s context limit. Suppose the application reserves 2,000 tokens for output and 500 for a safety margin. It has 13,500 tokens available for the rest of the request.

The input contains more than the user’s question. System and developer instructions, tool schemas, conversation history, retrieved passages, and message-format overhead all consume capacity. If instructions and tool definitions take 2,000 tokens and the current question takes 500, then 11,000 remain for history and retrieved evidence under the chosen budget.

Category	Reserved tokens	Policy
Instructions and tools	2,000	Preserve required behavior and available schemas
Current question	500	Reject or explicitly shorten oversized input
History	3,000	Keep facts needed for the current request
Retrieved evidence	8,000	Select passages with their conditions
Output	2,000	Leave room for a complete answer
Margin	500	Cover estimation and packaging differences

The categories sum to 16,000. The values are deliberately simple so you can audit the arithmetic. In production, reservations should follow measured request distributions and the model’s documented accounting rules.

View full-size image ↗

This allocation makes a tradeoff visible: retaining another 2,000 tokens of history means removing evidence, reducing output space, or rejecting the request. A hidden truncation function makes the same choice without telling the developer which requirement it sacrificed.

Count multilingual and structured inputs

Test the tokenizer on actual content types your audience will use. Include English, Vietnamese, code, URLs, numeric tables, and escaped JSON. Even if the assistant answers only in English, retrieved documents or user names can contain other scripts. Avoid assuming that a word count predicts their token count reliably.

A simple local inspection can record both the encoding and the input hash alongside the count. This makes a surprising budget change easier to reproduce after a tokenizer-library upgrade. You do not need to store sensitive input text in a general log merely to remember how large it was.

The provider’s usage record can include accounting details that a local plain-text count misses. Reconcile a few representative calls against those records. A local estimator is most useful for making preflight decisions; the returned usage record helps explain what the service actually processed and billed.

Run a count that another developer can reproduce

The following inspection fixes an encoding explicitly. It does not assume that every current model uses that encoding. These short strings are synthetic examples, so printing their counts does not expose customer content.

The example was checked with Python 3.12 and tiktoken==0.12.0 in an isolated environment on September 20, 2026. Install that package version in your own project environment to reproduce this snapshot, or record your chosen version when comparing an upgrade.

import tiktoken
from importlib.metadata import version

encoding = tiktoken.get_encoding("cl100k_base")
samples = [
    "The parcel is delayed.",
    "Kiện hàng đang bị chậm.",
    '{"estimated_delivery":null,"state":"delayed"}',
]
print(version("tiktoken"), encoding.name)
for sample in samples:
    tokens = encoding.encode(sample)
    assert encoding.decode(tokens) == sample
    print(len(sample), len(tokens))

That local run produced:

0.12.0 cl100k_base
22 5
23 11
45 11

The English and Vietnamese sentences have similar character counts but different token counts in this encoding. The JSON string has more characters yet the same token count as the Vietnamese example. These three observations support checking actual inputs; they do not establish a general token ratio for either language.

The first numeric column counts Python string characters; the second counts token IDs. Neither is a byte count, and neither includes a provider’s full message envelope. Run this with a recorded library version and preserve the literal inputs, including spaces and punctuation, when comparing results. A library or encoding change should be an explicit experiment rather than an unexplained production budget drift.

Karpathy’s minbpe, inspected on September 20, 2026, explains byte-level pair merging through short readable tokenizer implementations. That provides a useful reason to distrust a universal word-to-token ratio: token boundaries depend on the tokenizer’s learned merges and preprocessing. The repository is educational context, not authority for the token accounting of every provider or a guarantee that all architectures use the same algorithm.

Count the representation that will actually be sent. Escaping a string inside JSON changes that representation, while concatenating independently counted text pieces can also change token boundaries at their joins. Prefer an assembled-request estimate supported by the provider’s documented format. If the endpoint accepts image, audio or other non-text input, this plain-text snippet is insufficient for its accounting.

Preserve meaning when reducing context

Suppose the selected policy passage says “submit receipts within 14 days,” and the next paragraph defines an exception. Cutting the context exactly at the token limit can remove the exception while leaving a confident general rule. A document-aware selection policy should keep the conditions required to interpret a passage together.

Conversation history has similar dependencies. A user may clarify a country or product version in an earlier turn. Keeping only the last two messages could remove the fact that determines the correct answer. A summary can retain such facts, but it is another generated artifact that may omit or distort them. Preserve references to the original evidence where possible.

The prompt engineering guide, retrieved on September 20, 2026, discusses supplying relevant context and clear instructions. More context is not automatically more useful context. The application should choose material that supports the current task and fits its evidence contract.

Define the oversized-input response

A request can exceed the budget even after optional history is removed. If the required instructions, user input, and indispensable evidence do not fit, the application needs an explicit path. It can ask the user to narrow the question, split a document into separately evaluated tasks, or route to a model with an appropriate documented capacity.

Each choice changes behavior. Splitting a document can miss relationships across sections. A different model may have different structured-output support or quality. Asking the user to narrow the request adds friction but preserves a clear contract. Choose the path according to the product requirement, not merely whichever API call succeeds.

Avoid reducing the output reservation to almost zero to force an input through. An answer that stops before stating a limitation or completing a JSON object can create a different failure. The application should detect incomplete generation and avoid presenting a truncated result as a finished answer.

A useful pre-release fixture combines a long conversation, a multilingual question, and a policy exception near a chunk boundary. Verify the final assembled request, not just the individual pieces. Record which content was removed and why. Once you can explain that decision for one oversized request, you have a budget policy that can be tested instead of an unexplained string slice.

Make the boundary decision explicit

With the teaching budget above, 13,500 input tokens fit exactly after reserving the output and margin. An assembled estimate of 13,501 does not. The one-token difference should select a documented overflow path, not silently remove the end of a policy exception.

View full-size image ↗

For a boundary test, keep indispensable evidence fixed and enlarge only optional history. Verify that reducing that history preserves the current question’s country, product and date constraints. Next make the indispensable portion alone exceed capacity. The expected result is a clear refusal to assemble this request, followed by the chosen narrowing or routing action. The test should not merely assert that the final string fits; it should assert which evidence survived and which product behavior changed.

An application budget can be stricter than a provider’s maximum. That may be useful for controlling latency or keeping evidence selection manageable, but it remains a product choice that requires observation. The arithmetic here establishes capacity under stated reservations. It does not measure latency, predict an invoice or guarantee that the model will use every retained passage correctly.