Large Language Models vs Small Language Models

Small and large language models share transformer foundations but diverge sharply based on three constraints: deployment target, inference economics, and training budget. Architecturally, small models use grouped-query attention, sliding window attention, and shared KV caches to minimize memory footprint. Training-wise, they rely on high-quality synthetic data (e.g., Phi family), knowledge distillation from larger teachers (e.g., Gemma 2), and deliberate overtraining beyond compute-optimal ratios. Deployment involves quantization and hardware-specific tuning for devices like Apple's Neural Engine or NVIDIA Jetson. Small models have real gaps in generalization, multi-step reasoning, and world knowledge. Production systems increasingly compose both model classes using routing (small model handles easy requests, escalates hard ones), guardrails (small models filter input/output), and speculative decoding (small model drafts tokens, large model verifies). The key design insight is to start from constraints rather than benchmarks.