--- title: How we cut CI failure triage from hours to minutes using AI description: Red Hat's platform team built the Pipeline Failure Analyzer (PFA), an open-source tool that automates CI failure triage by restricting AI use to exactly one of... canonical: https://daily.dev/posts/how-we-cut-ci-failure-triage-from-hours-to-minutes-using-ai-bovfvbtfq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How we cut CI failure triage from hours to minutes using AI | daily.dev og:description: Red Hat's platform team built the Pipeline Failure Analyzer (PFA), an open-source tool that automates CI failure triage by restricting AI use to exactly one of... og:url: https://daily.dev/posts/how-we-cut-ci-failure-triage-from-hours-to-minutes-using-ai-bovfvbtfq og:image: https://api.daily.dev/og/posts/bOVfvbtFq.png og:image:alt: How we cut CI failure triage from hours to minutes using AI og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How we cut CI failure triage from hours to minutes using AI **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 13 min read · 0 upvotes · 0 comments ## Summary Red Hat's platform team built the Pipeline Failure Analyzer (PFA), an open-source tool that automates CI failure triage by restricting AI use to exactly one of five pipeline stages: root-cause analysis. The other four stages (log cleaning, report assembly, ticket creation, wiki publishing) run on deterministic Python. Raw trace logs are cleaned of ANSI codes and progress bars, then a two-tier regex system extracts error signals, compressing 10,000-line traces by 95-98%. An LLM agent then semantically groups failures across different hardware architectures that share no textual overlap (e.g., a missing PyTorch wheel producing different errors on x86_64 vs aarch64). Five validation layers guard against hallucinated AI output before tickets or reports are created, and a confidence-based system decides whether to comment on an existing ticket, create one with a review flag, or open a new one. Since deployment in June 2026, PFA diagnosed 110 pipeline failures with a 68% resolution rate and 78 auto-fixed merge requests. The post includes a minimal open-source log cleaner and error-extraction script developers can adopt without the full system. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you cluster CI failures across different CPU architectures that produce completely different error messages for the same root cause? Semantic grouping with an LLM agent solves this because traditional string-based clustering (edit distance, token overlap) fails when there's no text overlap between logs. For example, a missing PyTorch wheel causes 'Could not find a version that satisfies the requirement torch==2.1.0' on x86_64 but a cmake subprocess error on aarch64; an LLM agent can connect these as the same root cause where string matching cannot. _Developers wrestling with multi-architecture CI noise can find similar architecture breakdowns on daily.dev._ ### How much can you reduce a 10,000-line CI trace log before feeding it to an LLM for analysis? A deterministic cleaning and extraction process can compress a 10,000-line trace log to 200-500 lines, a 95-98% reduction, with virtually no signal loss. This works by stripping ANSI codes and progress bars, then capturing a window of 50 lines before and 20 lines after each regex-matched error anchor, merging overlapping windows and deduplicating the result. _Engineers optimizing log preprocessing before AI analysis can track techniques like this on daily.dev._ ### What validation checks should you run on AI-generated findings before acting on them in an automated pipeline? Five layers are needed: a per-finding verdict check for 13 required fields and enum membership, a per-group findings gate for section file cross-references, a per-reference path audit for canonical path format, a full JSON schema validation with cross-field checks, and a per-job trace check that rejects HTML error pages so the AI doesn't hallucinate a diagnosis from an API error response. _Teams building trust into AI-driven automation can compare validation approaches on daily.dev._ ## Similar posts on daily.dev - [Automating Root Cause Analysis for CI/CD Failures Using AI at Halodoc](https://daily.dev/posts/automating-root-cause-analysis-for-ci-cd-failures-using-ai-at-halodoc-tspinvhis) · Halodoc · 5 upvotes · 0 comments - [A Look Inside Our Auto-Reliability Agent Pipeline](https://daily.dev/posts/a-look-inside-our-auto-reliability-agent-pipeline-yh6hyclks) · Mabl Engineering Blog · 0 upvotes · 0 comments - [Your AI pipeline is broken, and your dashboards don’t know it](https://daily.dev/posts/your-ai-pipeline-is-broken-and-your-dashboards-don-t-know-it-7jrtsfsax) · The New Stack · 1 upvotes · 0 comments - [How I Built an AI-Powered “On-Call Engineer” That Likely Knows Why My Pipeline Broke](https://daily.dev/posts/how-i-built-an-ai-powered-on-call-engineer-that-likely-knows-why-my-pipeline-broke-6mf5acyrm) · Snowflake Community · 1 upvotes · 0 comments - [From Chaos to Clarity: Building an AI-Powered Smart Reporting System \(Part 1\)](https://daily.dev/posts/from-chaos-to-clarity-building-an-ai-powered-smart-reporting-system-part-1--u5sla1iqc) · ITNEXT · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#devops](https://daily.dev/tags/devops), [#ai-agents](https://daily.dev/tags/ai-agents), [#cicd](https://daily.dev/tags/cicd), [#logging](https://daily.dev/tags/logging) [View this post on daily.dev](https://daily.dev/posts/how-we-cut-ci-failure-triage-from-hours-to-minutes-using-ai-bovfvbtfq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How we cut CI failure triage from hours to minutes using AI","url":"https://daily.dev/posts/how-we-cut-ci-failure-triage-from-hours-to-minutes-using-ai-bovfvbtfq","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-we-cut-ci-failure-triage-from-hours-to-minutes-using-ai-bovfvbtfq"},"datePublished":"2026-09-29T13:33:08.851Z","dateModified":"2026-09-29T13:33:37.029Z","description":"Red Hat's platform team built the Pipeline Failure Analyzer (PFA), an open-source tool that automates CI failure triage by restricting AI use to exactly one of...","image":"https://media.daily.dev/image/upload/s--qPvKM23u--/f_auto/v1722860399/public/Placeholder%2009","thumbnailUrl":"https://media.daily.dev/image/upload/s--qPvKM23u--/f_auto/v1722860399/public/Placeholder%2009","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-we-cut-ci-failure-triage-from-hours-to-minutes-using-ai-bovfvbtfq","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,devops,ai-agents,cicd,logging","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"How we cut CI failure triage from hours to minutes using AI"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-we-cut-ci-failure-triage-from-hours-to-minutes-using-ai-bovfvbtfq#faq","mainEntity":[{"@type":"Question","name":"How do you cluster CI failures across different CPU architectures that produce completely different error messages for the same root cause?","acceptedAnswer":{"@type":"Answer","text":"Semantic grouping with an LLM agent solves this because traditional string-based clustering (edit distance, token overlap) fails when there's no text overlap between logs. For example, a missing PyTorch wheel causes 'Could not find a version that satisfies the requirement torch==2.1.0' on x86_64 but a cmake subprocess error on aarch64; an LLM agent can connect these as the same root cause where string matching cannot. Developers wrestling with multi-architecture CI noise can find similar architecture breakdowns on daily.dev."}},{"@type":"Question","name":"How much can you reduce a 10,000-line CI trace log before feeding it to an LLM for analysis?","acceptedAnswer":{"@type":"Answer","text":"A deterministic cleaning and extraction process can compress a 10,000-line trace log to 200-500 lines, a 95-98% reduction, with virtually no signal loss. This works by stripping ANSI codes and progress bars, then capturing a window of 50 lines before and 20 lines after each regex-matched error anchor, merging overlapping windows and deduplicating the result. Engineers optimizing log preprocessing before AI analysis can track techniques like this on daily.dev."}},{"@type":"Question","name":"What validation checks should you run on AI-generated findings before acting on them in an automated pipeline?","acceptedAnswer":{"@type":"Answer","text":"Five layers are needed: a per-finding verdict check for 13 required fields and enum membership, a per-group findings gate for section file cross-references, a per-reference path audit for canonical path format, a full JSON schema validation with cross-field checks, and a per-job trace check that rejects HTML error pages so the AI doesn't hallucinate a diagnosis from an API error response. Teams building trust into AI-driven automation can compare validation approaches on daily.dev."}}]} ```