What Is Tokenization in LLMs and Why AI Apps Depend on It

Tokenization is the process that turns raw text into the integer tokens a large language model actually reads. Models do not consume characters or words directly; a tokenizer maps strings into a sequence of IDs from a fixed vocabulary, and the model predicts the next ID in that sequence.

If you build LLM apps, tokenization sits under every cost estimate, context-window limit, chunking strategy, and evaluation quirk. Two prompts that look the same length in English can tokenize to very different counts—especially once you mix code, non-English text, or unusual Unicode. Understanding tokenization is different from only knowing that APIs bill by the token as a pricing unit: here the focus is how text becomes those units and why the mapping matters for product behavior.

What a tokenizer does

A tokenizer implements a reversible mapping between text and token IDs. Encode turns a string into a list of integers; decode turns those integers back into text. Modern LLM tokenizers usually work on subwords: pieces larger than a character and often smaller than a whole word. Common English text often lands near a few characters per token on average, but that is a rule of thumb, not a guarantee.

OpenAI’s open-source tiktoken library is a fast Byte Pair Encoding (BPE) tokenizer used with many of its models. Their docs highlight useful BPE properties: the mapping is reversible and lossless for the encoding, it works on arbitrary text, it compresses sequences relative to raw bytes, and it tends to reuse common subwords (for example endings like “ing”) so the model sees familiar pieces across contexts.

Byte Pair Encoding in one picture

BPE starts from a fine-grained base (often bytes) and repeatedly merges the most frequent adjacent pairs during training until it reaches a target vocabulary size. At inference time, encoding replays those merges in learned order: split the input, merge the highest-priority pairs that appear, and stop when nothing more merges. The vocabulary is essentially the ordered list of merges plus special tokens.

Why subwords? Whole-word vocabularies explode and break on new words. Character vocabularies stay small but make sequences very long, which is expensive for attention. Subword BPE is the compromise: short enough sequences, bounded vocabulary, and a way to represent rare or new strings by falling back to smaller pieces (even individual bytes).

Why tokenization matters for AI apps

• Context windows: Limits are counted in tokens, not words. Filling the window with verbose system prompts leaves less room for retrieved documents. See also why LLM apps hit token limits.

• Cost and latency: Input and output tokens drive billable usage and often latency. Counting tokens before you call the API prevents surprises.

• Chunking for RAG: Retrieval pipelines split documents by approximate token budgets so embeddings and prompts stay inside limits. Token-aware chunking beats naive character splits when languages or code density vary.

• Prompt fragility: Leading spaces, newlines, and special marker strings can map to different tokens than you expect. Small template edits can change model behavior because the token sequence changed.

• Multilingual and code text: Some scripts and symbols need more tokens per visible character than English prose, which affects both cost and how much content fits in context.

How to count tokens in practice

Match the tokenizer to the model family. With tiktoken you typically call encoding_for_model(...) or load a named encoding such as cl100k_base or o200k_base, then use len(encoding.encode(text)). OpenAI’s cookbook on how to count tokens with tiktoken walks through encodings and counting helpers. For interactive exploration, their tokenizer tool on the developer site visualizes how a string splits.

When you use another provider, do not assume OpenAI token counts transfer one-to-one. Always use that provider’s tokenizer (or documented approximation) for hard limits and billing estimates.

Tokenization vs embeddings (do not confuse them)

Tokenization produces discrete IDs for the model’s input layer. Embeddings turn text (often after tokenization inside the embedding model) into dense vectors for similarity search. You need both in RAG systems: tokenize/count to stay inside prompt budgets; embed to retrieve the right chunks.

FAQ: Is a token the same as a word?

No. A token can be a whole word, part of a word, punctuation, a space-attached fragment, or even a single byte for rare characters. English averages are handy for rough planning; exact counts require the real tokenizer.

FAQ: Why did my short prompt use so many tokens?

System instructions, tool schemas, chat history, and JSON wrappers all tokenize. Non-English text and dense code often use more tokens per character. Count the full request payload, not only the user sentence.

FAQ: Do I need to train my own tokenizer?

Almost never for application work. Use the tokenizer shipped with the model you call. Custom tokenizers appear mainly when teams train or heavily adapt base models—an advanced path beyond typical API integration.

Learn to build LLM apps with live practice

If you want hands-on practice with LLM applications, retrieval, and evaluation, guided by practitioners, Coderhouse offers live online courses with one class per week:

• AI Engineering Course: 8 weeks for experienced developers moving into AI engineering, covering LLM applications, retrieval, agents, evaluation, and deployment.

• Data Analytics Course: 11 weeks for career switchers, analysts, and data-driven professionals, covering metrics, SQL, Python, visualization, experimentation, and AI-assisted analysis.