Tokenisation

Tokenisation is the first step in how a language model processes text. Before a model can work with text, it must convert it into a sequence of numerical tokens — discrete units that the model's architecture can manipulate. A token might represent a whole word ("hello"), a sub-word ("un" + "likely"), a single character, or even a punctuation mark. The specific mapping between text and tokens is defined by the model's tokeniser, which is trained alongside the model. Different models use different tokenisers, which means the same text can be split into different tokens depending on the model being used. Understanding tokenisation is important because tokens directly determine context limits, costs, and processing speed.

Most modern language models use subword tokenisation algorithms like Byte-Pair Encoding (BPE) or SentencePiece. These algorithms learn to split text into a vocabulary of common subword units. Frequently used words remain as single tokens ("the", "and"), while rare or complex words are split into multiple tokens ("tokenisation" might become "token" + "isation"). A typical tokeniser vocabulary contains 30,000 to 100,000 tokens. The vocabulary size represents a trade-off: larger vocabularies mean fewer tokens per text (more efficient) but require more model parameters. Smaller vocabularies are more compact but produce longer token sequences. As a rough guide, one token corresponds to approximately 3-4 characters of English text, or about 75% of a word. A 1,000-word document is typically around 1,300-1,500 tokens, though this varies significantly by language and content type. Code, technical text, and non-English languages often require more tokens per word.

Tokenisation directly impacts three critical aspects of AI deployment: cost, context limits, and performance. Most AI API providers charge per token, so understanding how your content tokenises is essential for cost forecasting. A document that tokenises into twice as many tokens costs twice as much to process. Context windows are measured in tokens, not words. A model with a 128,000-token context window can process roughly 96,000 words — but this varies based on the content. Understanding this relationship is important for designing applications that stay within context limits. Tokenisation quality also affects model performance. Models perform better on text that tokenises efficiently (common words, standard English) and may struggle with heavily tokenised content (unusual terminology, non-Latin scripts, code in uncommon languages). This has practical implications for multilingual applications and specialised domains.

When building AI applications, tokenisation considerations include estimating API costs based on average token counts per request, designing prompts that are token-efficient without sacrificing clarity, chunking documents appropriately for RAG systems (splitting at meaningful token boundaries), and testing how domain-specific terminology tokenises in your chosen model. Tools like OpenAI's tiktoken library allow developers to count tokens before making API calls, enabling precise cost estimation and context management. Most AI development frameworks provide similar tokenisation utilities for their respective models.