--- title: How ChatGPT Optimizes its Agent Loop: Harness, API, and... description: OpenAI engineers share the efficiency techniques behind Codex and ChatGPT Work across three layers of their agentic AI stack. At the harness layer: persistent... canonical: https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference | daily.dev og:description: OpenAI engineers share the efficiency techniques behind Codex and ChatGPT Work across three layers of their agentic AI stack. At the harness layer: persistent... og:url: https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v og:image: https://api.daily.dev/og/posts/tubTAUM7v.png og:image:alt: How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference **[ByteByteGo](https://daily.dev/sources/bytebytego)** · 23 min read · 6 upvotes · 1 comments ## Summary OpenAI engineers share the efficiency techniques behind Codex and ChatGPT Work across three layers of their agentic AI stack. At the harness layer: persistent WebSockets eliminate repeated TCP/TLS handshakes, stable prompt prefixes preserve KV cache hits, deferred tool discovery keeps prompts lean, and Code Mode batches tool calls into a single program. At the API layer: incremental tokenization processes only new delta tokens instead of the full conversation, and safety classifiers run in parallel with inference to hide latency. At the inference layer: cache-aware routing sends requests back to machines holding their KV cache, intelligent eviction policies manage cache memory, speculative decoding uses a small draft model to propose tokens verified in one parallel pass, and prefill/decode separation lets each phase run on hardware optimized for its workload. The overarching lesson is that compounding many small optimizations across all layers — rather than focusing on any single technique — drives the biggest cost reductions. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Codex avoid resending the entire conversation history on every tool call? Codex keeps a persistent WebSocket connection open across all model calls in a turn instead of opening a new HTTPS connection each time. After a tool call, it sends only the new tool result along with a previous_response_id reference, and the server reconstructs the full context from state it already stored, avoiding repeated TCP/TLS handshakes and payload resending. _Engineers debugging agent latency compare architecture patterns like this one on daily.dev._ ### What is speculative decoding and how does it speed up LLM token generation? Speculative decoding uses a small, fast draft model to propose several upcoming tokens at once, which the large model then verifies in a single parallel pass instead of generating each token sequentially. Correct proposals are accepted without extra cost, while wrong ones are discarded and regenerated by the large model, so output quality is unchanged. The average acceptance length measures how well the draft model performs. _Developers weighing inference optimization techniques track deep dives like this on daily.dev._ ### Why does changing the order of tool definitions in an LLM prompt increase inference cost? Prompt caching only reuses computation when the prefix matches the previous request byte-for-byte and token-by-token; any difference near the front forces recomputation of everything after it. Codex once stored MCP tool definitions in a hash map without guaranteed ordering, causing the same tools to serialize in different orders across requests, silently breaking the cache and raising cost without changing correctness. _Teams tuning prompt caching strategies keep track of gotchas like this via daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@rizzdev** · 1 upvotes > Engineers can stack those cache and routing tricks. Model selection upstream still sets the ceiling those tricks cannot raise ## Similar posts on daily.dev - [GPT-5.6 kernel of truth: Sol can cut its own costs, says OpenAI](https://daily.dev/posts/gpt-5-6-kernel-of-truth-sol-can-cut-its-own-costs-says-openai-q2dfgmfke) · The New Stack · 4 upvotes · 1 comments - [Streaming Tokens and Tools: Multi-Turn Agentic Harness Support in NVIDIA Dynamo](https://daily.dev/posts/streaming-tokens-and-tools-multi-turn-agentic-harness-support-in-nvidia-dynamo-5rhbl5ps2) · NVIDIA Developer · 1 upvotes · 0 comments - [Improving Deep Agents with harness engineering](https://daily.dev/posts/improving-deep-agents-with-harness-engineering-czhznhb88) · LangChain · 1 upvotes · 0 comments - [Codex from 0 to 10M Users: Building ChatGPT Work — Akshay Nathan, OpenAI](https://daily.dev/posts/codex-from-0-to-10m-users-building-chatgpt-work-akshay-nathan-openai-v6rnymbfw) · Latent Space · 0 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#chatgpt](https://daily.dev/tags/chatgpt), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference","url":"https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v"},"datePublished":"2026-07-29T15:20:27.828Z","dateModified":"2026-09-14T06:40:34.298Z","description":"OpenAI engineers share the efficiency techniques behind Codex and ChatGPT Work across three layers of their agentic AI stack. At the harness layer: persistent...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3c1c65734cf72de2740e801d2b14cc61?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3c1c65734cf72de2740e801d2b14cc61?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ByteByteGo","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ByteByteGo","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/35be29234ee14d01a9cd049c52e12753","url":"https://daily.dev/sources/bytebytego"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":6},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"ai-agents,chatgpt,ai-inference","timeRequired":"PT23M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ByteByteGo","item":"https://daily.dev/sources/bytebytego"},{"@type":"ListItem","position":3,"name":"How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v","comment":[{"@type":"Comment","text":"Engineers can stack those cache and routing tricks. Model selection upstream still sets the ceiling those tricks cannot raise","datePublished":"2026-07-31T16:05:30.195Z","url":"https://daily.dev/posts/tubTAUM7v#c-hTUPtZQf2","author":{"@type":"Person","name":"Andrew","url":"https://daily.dev/rizzdev","image":"https://media.daily.dev/image/upload/s--35vRfXAA--/f_auto/v1785413727/avatars/avatar_fQ7ttCKtPuu6tDvcgQXv5?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-chatgpt-optimizes-its-agent-loop-harness-api-and-inference-tubtaum7v#faq","mainEntity":[{"@type":"Question","name":"How does Codex avoid resending the entire conversation history on every tool call?","acceptedAnswer":{"@type":"Answer","text":"Codex keeps a persistent WebSocket connection open across all model calls in a turn instead of opening a new HTTPS connection each time. After a tool call, it sends only the new tool result along with a previous_response_id reference, and the server reconstructs the full context from state it already stored, avoiding repeated TCP/TLS handshakes and payload resending. Engineers debugging agent latency compare architecture patterns like this one on daily.dev."}},{"@type":"Question","name":"What is speculative decoding and how does it speed up LLM token generation?","acceptedAnswer":{"@type":"Answer","text":"Speculative decoding uses a small, fast draft model to propose several upcoming tokens at once, which the large model then verifies in a single parallel pass instead of generating each token sequentially. Correct proposals are accepted without extra cost, while wrong ones are discarded and regenerated by the large model, so output quality is unchanged. The average acceptance length measures how well the draft model performs. Developers weighing inference optimization techniques track deep dives like this on daily.dev."}},{"@type":"Question","name":"Why does changing the order of tool definitions in an LLM prompt increase inference cost?","acceptedAnswer":{"@type":"Answer","text":"Prompt caching only reuses computation when the prefix matches the previous request byte-for-byte and token-by-token; any difference near the front forces recomputation of everything after it. Codex once stored MCP tool definitions in a hash map without guaranteed ordering, causing the same tools to serialize in different orders across requests, silently breaking the cache and raising cost without changing correctness. Teams tuning prompt caching strategies keep track of gotchas like this via daily.dev."}}]} ```