How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference

OpenAI engineers share the efficiency techniques behind Codex and ChatGPT Work across three layers of their agentic AI stack. At the harness layer: persistent WebSockets eliminate repeated TCP/TLS handshakes, stable prompt prefixes preserve KV cache hits, deferred tool discovery keeps prompts lean, and Code Mode batches tool calls into a single program. At the API layer: incremental tokenization processes only new delta tokens instead of the full conversation, and safety classifiers run in parallel with inference to hide latency. At the inference layer: cache-aware routing sends requests back to machines holding their KV cache, intelligent eviction policies manage cache memory, speculative decoding uses a small draft model to propose tokens verified in one parallel pass, and prefill/decode separation lets each phase run on hardware optimized for its workload. The overarching lesson is that compounding many small optimizations across all layers — rather than focusing on any single technique — drives the biggest cost reductions.