Core Concepts

Quick definitions for the vocabulary you'll run into everywhere. Each one is short, and several get a fuller treatment later in the handbook.

Context windows & tokenization. The context window is the maximum number of tokens (prompt + generated output) a model can attend to at once. Frontier models in 2026 commonly offer 100K to 1M+ token windows 🔄. But bigger isn't automatically better. Attention cost grows with length, and models exhibit a well-documented "lost in the middle" effect where information buried in the center of a long context is recalled less reliably than material at the start or end. Treat context as a scarce, curated resource, not a dumping ground.

Parameters vs. active params / MoE. "Parameters" are the learned weights, the crude proxy for model capacity. A Mixture-of-Experts (MoE) model has many parameters but only activates a subset (a few "experts") per token, so a model with, say, hundreds of billions of total parameters might use only tens of billions per forward pass. So MoE decouples total capacity (which helps quality) from per-token compute (which drives cost and speed). This is why you'll see "total parameters" and "active parameters" quoted separately. The latter is what you actually pay for at inference.

Quantization & precision. Weights are usually trained in 16-bit precision but can be quantized down to 8-bit, 4-bit, or lower for inference. This shrinks memory footprint and speeds things up, at some cost to quality. A 4-bit quant of a big model often beats a full-precision small model at the same memory budget. If you're running models locally or self-hosting, quantization is the single biggest lever on what fits your hardware.

Reasoning / test-time compute. Covered in How LLMs Actually Work. The mental model is to spend more tokens thinking to buy accuracy on hard, decomposable problems. Not worth it for simple lookups, formatting, or classification, and you'll pay latency for nothing.

Multimodality. Modern models increasingly ingest (and sometimes emit) more than text, including images, audio, and video, by encoding each modality into the same token stream the language model already understands. In practice, you can hand a model a screenshot, a chart, or a PDF page and ask about it. Native multimodality (trained in from the start) tends to reason across modalities better than bolted-on adapters.

Embeddings & vector search. An embedding is a fixed-length vector that captures the meaning of a piece of text (or image), such that semantically similar things land near each other in vector space. Store a corpus of embeddings in a vector database and you can retrieve by meaning rather than keywords, embedding the query and finding its nearest neighbors. This is the retrieval engine under most RAG systems. Full treatment in RAG & Retrieval.

RAG. Retrieval-Augmented Generation grounds a model's answer in documents you fetch at query time (usually via embedding search) and inject into the context window, instead of relying on what it memorized during training. It's the standard way to make a model use your private or current data without retraining it. Full treatment in RAG & Retrieval.

Fine-tuning vs. prompting vs. tools. Three ways to change model behavior, in rough order of effort:

• Prompting. Change what you say. Zero cost, instant, the right first move for most problems.
• Tools. Give the model functions to call (search, code execution, APIs) so it can fetch facts and take actions instead of hallucinating them. The backbone of agents.
• Fine-tuning. Change the weights on your own data. Powerful for style, format, and narrow domains, but costly, slow to iterate, and often unnecessary once prompting + RAG + tools are exhausted. See Fine-Tuning & Post-Training. The practical heuristic: reach for fine-tuning last, not first.

Prompt caching. Many providers let you cache a large, static prefix of your prompt (a system prompt, a big document, a tool schema) so that reusing it across requests is much cheaper and faster than re-processing it every time. If your app sends the same 10K-token preamble on every call, caching it can cut cost and latency dramatically. Design your prompts so the stable parts come first. That's what gets cached.

Structured output. Instead of free-form prose, you can constrain a model to emit valid JSON matching a schema (via JSON mode, function/tool schemas, or grammar-constrained decoding). This is what makes LLMs usable as reliable components in software. You get parseable data, not a paragraph you have to regex. When an LLM output feeds another program, prefer structured output. Patterns in Prompting & Context Engineering.