--- title: 8 engineering lessons on running AI agents in production description: Drawing on a Redpanda-hosted conversation with Red Canary co-founder Brian Beyer and furl CTO Joe Moles, this piece distills eight lessons for putting AI... canonical: https://daily.dev/posts/8-engineering-lessons-on-running-ai-agents-in-production-utaezdznu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: 8 engineering lessons on running AI agents in production | daily.dev og:description: Drawing on a Redpanda-hosted conversation with Red Canary co-founder Brian Beyer and furl CTO Joe Moles, this piece distills eight lessons for putting AI... og:url: https://daily.dev/posts/8-engineering-lessons-on-running-ai-agents-in-production-utaezdznu og:image: https://api.daily.dev/og/posts/UtaEZDzNU.png og:image:alt: 8 engineering lessons on running AI agents in production og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 8 engineering lessons on running AI agents in production **[Redpanda](https://daily.dev/sources/redpanda)** · 13 min read · 0 upvotes · 0 comments ## Summary Drawing on a Redpanda-hosted conversation with Red Canary co-founder Brian Beyer and furl CTO Joe Moles, this piece distills eight lessons for putting AI agents into production based on their experience building a large-scale security data platform. Key ideas include working backward from the desired outcome rather than the data, decomposing systems into simple single-purpose components to make agent experiments cheap, favoring deterministic code over agents wherever logic is scriptable, treating agents like new hires with defined roles, KPIs, and access scoping, relying on clean human-decided exemplar data for tuning, and documenting existing processes (even a thin slice) as the starting point when a team lacks that foundation. The piece ends with a soft pitch for Redpanda's Agentic Data Plane. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Should I use an AI agent or write deterministic code for a given task in production? Write deterministic code first whenever the logic is reliably rule-based (A plus B equals C), since it is cheaper, faster, and simpler to reason about. Reserve agents for work that is well understood conceptually but not fully scriptable, such as knowing which 20-30 questions to ask about a security event and having the agent investigate and summarize the answer. _Weigh deterministic code versus agentic approaches for your own workflows with engineering write-ups curated on daily.dev._ ### How do you decide what data access to give an AI agent in production? Apply the same test used for a new hire: ask whether you would give a newly hired engineer or analyst access to all of your production data. If the answer is no, the agent should not get that access either, and access needs to be deliberately scoped and enforced, similar to vetting new employees as a team grows larger and more international. _Developers scoping agent permissions can track access-control practices for agentic systems on daily.dev._ ### What should teams do if they want to build AI agents but lack years of documented decisions or clean training data? Start by writing down what you already do, beginning with a thin slice of a single process rather than trying to document an entire workflow at once. This surfaces steps you hadn't consciously thought of, which can then be documented too, and a diagram showing data flow and decision points for human readers is the key artifact, since agentic systems can't yet infer architecture intent on their own. _Teams building an agent foundation from scratch can follow practical playbooks like this one on daily.dev._ ## Similar posts on daily.dev - [4 FAQs about designing agentic systems for production](https://daily.dev/posts/4-faqs-about-designing-agentic-systems-for-production-cxf1auxcm) · Redpanda · 0 upvotes · 0 comments - [Five principles for governed autonomy with enterprise AI](https://daily.dev/posts/five-principles-for-governed-autonomy-with-enterprise-ai-telaphuim) · Redpanda · 0 upvotes · 0 comments - [Building and Deploying AI Agents in Production: Lessons I Learned](https://daily.dev/posts/building-and-deploying-ai-agents-in-production-lessons-i-learned-fmxhggdbm) · Medium · 1 upvotes · 0 comments - [Governing AI agents in production: Redpanda Agentic Data Plane](https://daily.dev/posts/governing-ai-agents-in-production-redpanda-agentic-data-plane-riajexr7p) · Redpanda · 0 upvotes · 0 comments --- Tags: [#security](https://daily.dev/tags/security), [#career](https://daily.dev/tags/career), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/8-engineering-lessons-on-running-ai-agents-in-production-utaezdznu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"8 engineering lessons on running AI agents in production","url":"https://daily.dev/posts/8-engineering-lessons-on-running-ai-agents-in-production-utaezdznu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/8-engineering-lessons-on-running-ai-agents-in-production-utaezdznu"},"datePublished":"2026-09-23T03:00:05.695Z","dateModified":"2026-09-23T03:00:33.306Z","description":"Drawing on a Redpanda-hosted conversation with Red Canary co-founder Brian Beyer and furl CTO Joe Moles, this piece distills eight lessons for putting AI...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/591623527254bf8fd52697f24078712e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/591623527254bf8fd52697f24078712e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Redpanda","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Redpanda","logo":"https://media.daily.dev/image/upload/s--1R2EIDcb--/f_auto,q_auto/v1769934743/logos/redpanda","url":"https://daily.dev/sources/redpanda"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/8-engineering-lessons-on-running-ai-agents-in-production-utaezdznu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"security,career,ai-agents","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Redpanda","item":"https://daily.dev/sources/redpanda"},{"@type":"ListItem","position":3,"name":"8 engineering lessons on running AI agents in production"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/8-engineering-lessons-on-running-ai-agents-in-production-utaezdznu#faq","mainEntity":[{"@type":"Question","name":"Should I use an AI agent or write deterministic code for a given task in production?","acceptedAnswer":{"@type":"Answer","text":"Write deterministic code first whenever the logic is reliably rule-based (A plus B equals C), since it is cheaper, faster, and simpler to reason about. Reserve agents for work that is well understood conceptually but not fully scriptable, such as knowing which 20-30 questions to ask about a security event and having the agent investigate and summarize the answer. Weigh deterministic code versus agentic approaches for your own workflows with engineering write-ups curated on daily.dev."}},{"@type":"Question","name":"How do you decide what data access to give an AI agent in production?","acceptedAnswer":{"@type":"Answer","text":"Apply the same test used for a new hire: ask whether you would give a newly hired engineer or analyst access to all of your production data. If the answer is no, the agent should not get that access either, and access needs to be deliberately scoped and enforced, similar to vetting new employees as a team grows larger and more international. Developers scoping agent permissions can track access-control practices for agentic systems on daily.dev."}},{"@type":"Question","name":"What should teams do if they want to build AI agents but lack years of documented decisions or clean training data?","acceptedAnswer":{"@type":"Answer","text":"Start by writing down what you already do, beginning with a thin slice of a single process rather than trying to document an entire workflow at once. This surfaces steps you hadn't consciously thought of, which can then be documented too, and a diagram showing data flow and decision points for human readers is the key artifact, since agentic systems can't yet infer architecture intent on their own. Teams building an agent foundation from scratch can follow practical playbooks like this one on daily.dev."}}]} ```