--- title: Tracing failures across Terraform state boundaries with... description: A security group egress rule change in one Terraform stack can silently break a Lambda function's connection to DynamoDB in another stack, and AWS DevOps Agent... canonical: https://daily.dev/posts/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent-qivphpgmb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Tracing failures across Terraform state boundaries with AWS DevOps Agent | daily.dev og:description: A security group egress rule change in one Terraform stack can silently break a Lambda function's connection to DynamoDB in another stack, and AWS DevOps Agent... og:url: https://daily.dev/posts/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent-qivphpgmb og:image: https://api.daily.dev/og/posts/qivpHPGMb.png og:image:alt: Tracing failures across Terraform state boundaries with AWS DevOps Agent og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Tracing failures across Terraform state boundaries with AWS DevOps Agent **[AWS](https://daily.dev/sources/aws)** · 18 min read · 0 upvotes · 0 comments ## Summary A security group egress rule change in one Terraform stack can silently break a Lambda function's connection to DynamoDB in another stack, and AWS DevOps Agent helps trace that failure across the state boundary. In the demo, an infrastructure team tightens an egress rule, severing the app team's Lambda from DynamoDB through a Gateway VPC endpoint. CloudWatch alarms fire, but the Lambda logs show timeouts with no clear networking cause, and the DynamoDB table looks healthy, misleading a human investigator. AWS DevOps Agent correlates the alarms, maps the dependency chain across stacks, and uses CloudTrail to pinpoint the egress change as root cause, roughly a minute before the timeouts began. It then produces a mitigation spec that Kiro turns into a reviewed Terraform diff to restore the rule. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why would a Lambda function time out connecting to DynamoDB even though IAM permissions and the DynamoDB table both look healthy? A security group egress rule change in a separate networking Terraform stack blocked the Lambda function's outbound path to the DynamoDB Gateway VPC endpoint on TCP/443. Lambda logs show only a generic 30-second timeout with no AccessDenied error, and DynamoDB metrics show near-zero consumed capacity because requests never reach the table, making the real cause easy to miss. _Debugging a cross-stack Terraform networking issue? Track tools like AWS DevOps Agent and Kiro on daily.dev._ ### How do I provision AWS DevOps Agent resources with Terraform since they're not in the standard hashicorp/aws provider? AWS DevOps Agent resources, including the Agent Space and account association, are provisioned through the hashicorp/awscc provider (the AWS Cloud Control API provider), not hashicorp/aws. The IAM trust policy must use the service principal aidevops.amazonaws.com with an aws:SourceArn condition set to the exact Agent Space ARN; a wildcard ARN causes a 400 error. Terraform 1.6.0+ with awscc 1.66.0+ is required. _Setting up agent-based AWS tooling with Terraform? Follow Terraform and AWS updates on daily.dev._ ### How does AWS DevOps Agent correlate CloudWatch alarms with CloudTrail to find the root cause of an outage? It fans investigation work into parallel subagents examining Lambda metrics, DynamoDB metrics, Lambda logs, CloudTrail events, and VPC networking simultaneously, then correlates alarms that started in the same time window as a single event. It pulls recent CloudTrail events for resources in its topology map and matches a security group change's timestamp against the onset of Lambda timeouts, roughly a minute later, to establish causation. _Evaluating AI-driven incident investigation tools? Follow AWS DevOps Agent coverage on daily.dev._ ## Similar posts on daily.dev - [Automate root cause analysis across Datadog and Elasticsearch with AWS DevOps Agent](https://daily.dev/posts/automate-root-cause-analysis-across-datadog-and-elasticsearch-with-aws-devops-agent-pg6b3fzvz) · AWS · 0 upvotes · 0 comments - [Integration with AWS DevOps Agent: Autonomous investigations powered by production context](https://daily.dev/posts/integration-with-aws-devops-agent-autonomous-investigations-powered-by-production-context-bne3y8fst) · Dynatrace · 4 upvotes · 0 comments - [Integrate AWS DevOps Agent with third-party tools using Amazon EventBridge](https://daily.dev/posts/integrate-aws-devops-agent-with-third-party-tools-using-amazon-eventbridge-zzl8cra6e) · AWS · 0 upvotes · 0 comments - [AWS DevOps Agent Meets Chaos How I Wired an Autonomous SRE to a Deliberately Broken EKS Cluster](https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q) · Medium · 1 upvotes · 0 comments - [AWS Announces General Availability of DevOps Agent for Automated Incident Investigation](https://daily.dev/posts/aws-announces-general-availability-of-devops-agent-for-automated-incident-investigation-o3foudjfb) · InfoQ · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#terraform](https://daily.dev/tags/terraform), [#kiro](https://daily.dev/tags/kiro) [View this post on daily.dev](https://daily.dev/posts/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent-qivphpgmb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Tracing failures across Terraform state boundaries with AWS DevOps Agent","url":"https://daily.dev/posts/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent-qivphpgmb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent-qivphpgmb"},"datePublished":"2026-10-26T17:25:30.359Z","dateModified":"2026-10-26T17:32:14.523Z","description":"A security group egress rule change in one Terraform stack can silently break a Lambda function's connection to DynamoDB in another stack, and AWS DevOps Agent...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c54ea358b85c401e223f2db513f72f8b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c54ea358b85c401e223f2db513f72f8b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"AWS","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AWS","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/aws","url":"https://daily.dev/sources/aws"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent-qivphpgmb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,terraform,kiro","timeRequired":"PT18M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AWS","item":"https://daily.dev/sources/aws"},{"@type":"ListItem","position":3,"name":"Tracing failures across Terraform state boundaries with AWS DevOps Agent"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/tracing-failures-across-terraform-state-boundaries-with-aws-devops-agent-qivphpgmb#faq","mainEntity":[{"@type":"Question","name":"Why would a Lambda function time out connecting to DynamoDB even though IAM permissions and the DynamoDB table both look healthy?","acceptedAnswer":{"@type":"Answer","text":"A security group egress rule change in a separate networking Terraform stack blocked the Lambda function's outbound path to the DynamoDB Gateway VPC endpoint on TCP/443. Lambda logs show only a generic 30-second timeout with no AccessDenied error, and DynamoDB metrics show near-zero consumed capacity because requests never reach the table, making the real cause easy to miss. Debugging a cross-stack Terraform networking issue? Track tools like AWS DevOps Agent and Kiro on daily.dev."}},{"@type":"Question","name":"How do I provision AWS DevOps Agent resources with Terraform since they're not in the standard hashicorp/aws provider?","acceptedAnswer":{"@type":"Answer","text":"AWS DevOps Agent resources, including the Agent Space and account association, are provisioned through the hashicorp/awscc provider (the AWS Cloud Control API provider), not hashicorp/aws. The IAM trust policy must use the service principal aidevops.amazonaws.com with an aws:SourceArn condition set to the exact Agent Space ARN; a wildcard ARN causes a 400 error. Terraform 1.6.0+ with awscc 1.66.0+ is required. Setting up agent-based AWS tooling with Terraform? Follow Terraform and AWS updates on daily.dev."}},{"@type":"Question","name":"How does AWS DevOps Agent correlate CloudWatch alarms with CloudTrail to find the root cause of an outage?","acceptedAnswer":{"@type":"Answer","text":"It fans investigation work into parallel subagents examining Lambda metrics, DynamoDB metrics, Lambda logs, CloudTrail events, and VPC networking simultaneously, then correlates alarms that started in the same time window as a single event. It pulls recent CloudTrail events for resources in its topology map and matches a security group change's timestamp against the onset of Lambda timeouts, roughly a minute later, to establish causation. Evaluating AI-driven incident investigation tools? Follow AWS DevOps Agent coverage on daily.dev."}}]} ```