Tokens are the fundamental units AI models use to process text — they can represent full words, subwords, characters, or short phrases. Before any text reaches an LLM, a tokenizer breaks it into these segments and maps them to numerical IDs. Modern LLMs use subword tokenization to balance vocabulary size and computational efficiency. Token counts directly affect context windows (how much a model can 'remember'), inference costs (billed per input/output token), latency (models generate one token at a time), and output quality (the 'lost in the middle' problem). For developers, understanding tokens is critical for writing efficient prompts, chunking data for RAG pipelines, and forecasting infrastructure costs. Roughly 750 words equal 1,000 tokens, and different models tokenize the same text differently.
