A practical build log for an AI agent called Foodie that generates eating itineraries from a Telegram message, using the Hermes open-source framework and DigitalOcean's Serverless Inference Engine. The post focuses on two key performance metrics — time to first token (TTFT) and total generation time — and explains concrete techniques to improve each: trimming context size, parallelizing tool calls, sending a quick acknowledgment before heavy work, using small models for background tasks, and delivering text before audio renders. It also provides a clear framework for deciding when serverless inference is the right choice versus dedicated GPU hosting or fully self-hosted models, based on cost, data control, custom model needs, and latency floors.
Table of contents
The test systemSetup, brieflyThe product is two Markdown filesMaking it fastWhen to move off serverless inferenceWhat this build says about the stack604 Impressions1 Comment