Kimi K3 Architecture Notes

A concise architectural breakdown of Kimi K3, a 2.8T parameter open-weight model. Key highlights include: LatentMoE (compressing large linear layers similar to multi-head latent attention), Kimi Delta Attention for inference efficiency, attention residuals that connect residuals across layers with attention-weighted contributions (adding ~4% training cost and ~2% inference cost), full replacement of RoPE with NoPE (No Positional Embeddings) across all layers — a first for a frontier-level model — and native multimodal support. The model is positioned as a scaled-up production version of Kimi Linear (48B → 2.8T), following the broader industry trend toward inference-efficient architectures seen in Nemotron 3 and DeepSeek V4.