Halodoc reduced LLM API costs by 5–15% by switching from JSON to TOON (Token-Oriented Object Notation) for structured data in prompts. TOON eliminates JSON's syntactic overhead — braces, quoted keys, commas — and represents arrays of objects as CSV-style tables, cutting repeated key names. Evaluated against MessagePack, Protocol Buffers, and YAML, TOON offered the best balance of token efficiency (~35–45% theoretical reduction on structured data), LLM compatibility (~95% across OpenAI, Anthropic, Google, AWS Bedrock), and low integration effort. A shared Python SDK with JSON↔TOON converters was built, with config flags enabling per-use-case rollout and instant rollback. Output token optimization (tested at 8–43% reduction in POC) was not yet deployed to production. The post also covers complementary strategies: response caching with DynamoDB and batch API processing.
