<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq" -->

---
title: Context Windows Don’t Know What’s Still True — I Built a...
description: A deterministic, pure-Python benchmark explores why AI agents fail when they act on context that is technically present but no longer true. The author builds...
canonical: https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Context Windows Don’t Know What’s Still True — I Built a Validity Layer That Does | daily.dev
og:description: A deterministic, pure-Python benchmark explores why AI agents fail when they act on context that is technically present but no longer true. The author builds...
og:url: https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq
og:image: https://api.daily.dev/og/posts/tnQMIlYCq.png
og:image:alt: Context Windows Don’t Know What’s Still True — I Built a Validity Layer That Does
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Context Windows Don’t Know What’s Still True — I Built a Validity Layer That Does

**[Towards Data Science](https://daily.dev/sources/tds)** · 21 min read · 1 upvotes · 0 comments

## Summary

A deterministic, pure-Python benchmark explores why AI agents fail when they act on context that is technically present but no longer true. The author builds two executors — a baseline that only discovers broken dependencies when an action fails, and a validity-aware version that checks a fact's status (ACTIVE, STALE, SUPERSEDED, UNKNOWN) before acting. Across four experiments, results show validity-aware execution eliminates pre-failure wasted work, that a 96-configuration sweep found wasted work scales with the size of the affected dependency closure rather than graph shape, that verification checks add cost on false alarms but pay off on real changes, and that under tight step budgets the validity-aware executor completes tasks the baseline fails. Code and the full benchmark are open-sourced on GitHub.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://towardsdatascience.com/context-windows-dont-know-whats-still-true-i-built-a-validity-layer-that-does>

## Questions this post answers

### How much wasted work does an AI agent do before it notices a dependency became stale?

In a deterministic benchmark, pre-failure work equals the total affected dependency closure size minus one, confirmed across a 96-configuration sweep with zero exceptions. A baseline executor that only detects failures after acting wasted up to nine steps on a doomed plan, while a validity-aware executor that checks dependency status before acting produced zero pre-failure steps in every tested case.

_Anyone debugging agents that follow broken plans can find deeper context on tracking state validity via daily.dev._

### Does checking whether a fact is still valid before acting always help an AI agent?

No, verification pays off only when the underlying fact actually changed. When a suspicious signal turned out to be a false alarm, the validity-aware executor spent one extra step confirming something that was fine (4 steps versus 3 for a baseline that never checked), while on a genuine change checking saved a step (6 versus 7) by avoiding a doomed action entirely.

_daily.dev surfaces this kind of trade-off analysis for engineers weighing verification costs in agent design._

### Can wasted steps from stale context cause an otherwise solvable AI agent task to fail entirely?

Yes, when the step budget is tight. In a swept-budget test on a value-change scenario, a baseline executor failed at budgets of 6, 7, and 8 steps because it wasted two steps on a doomed plan before recovering, while a validity-aware executor completed the same task at those same budgets since it avoided the wasted steps.

_daily.dev helps engineers keep tabs on findings like this when designing resource-constrained agent workflows._

## Similar posts on daily.dev

- [Validating agentic behavior when “correct” isn’t deterministic](https://daily.dev/posts/validating-agentic-behavior-when-correct-isn-t-deterministic-clsrnjpoc) · GitHub Blog · 1 upvotes · 0 comments
- [AI Agents Don’t Need More Context — They Need Typed Context](https://daily.dev/posts/ai-agents-don-t-need-more-context-they-need-typed-context-216ngg9ih) · Towards Data Science · 5 upvotes · 2 comments
- [Your AI Agent Was Great in Week One. Here’s Why It’s Wrong by Month Six.](https://daily.dev/posts/your-ai-agent-was-great-in-week-one-here-s-why-it-s-wrong-by-month-six--soihggeax) · Medium · 0 upvotes · 0 comments
- [Google Newsletter Issue Agent-CLI Eval](https://daily.dev/posts/google-newsletter-issue-agent-cli-eval-zbtoft45i) · Daily Dose of Data Science \| Avi Chawla \| Substack · 3 upvotes · 1 comments
- [AI Reasoning Explained: Why Context Matters](https://daily.dev/posts/ai-reasoning-explained-why-context-matters-kfgm8uclg) · Redis · 1 upvotes · 0 comments

---

Tags: [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents)

[View this post on daily.dev](https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Context Windows Don’t Know What’s Still True — I Built a Validity Layer That Does","url":"https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq"},"datePublished":"2026-09-08T11:05:59.573Z","dateModified":"2026-09-13T19:10:49.747Z","description":"A deterministic, pure-Python benchmark explores why AI agents fail when they act on context that is technically present but no longer true. The author builds...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/38109a7fbcbf95b52cb64e4f67ee5fb0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/38109a7fbcbf95b52cb64e4f67ee5fb0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Towards Data Science","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Towards Data Science","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/tds","url":"https://daily.dev/sources/tds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,llm,ai-agents","timeRequired":"PT21M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Towards Data Science","item":"https://daily.dev/sources/tds"},{"@type":"ListItem","position":3,"name":"Context Windows Don’t Know What’s Still True — I Built a Validity Layer That Does"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/context-windows-don-t-know-what-s-still-true-i-built-a-validity-layer-that-does-tnqmilycq#faq","mainEntity":[{"@type":"Question","name":"How much wasted work does an AI agent do before it notices a dependency became stale?","acceptedAnswer":{"@type":"Answer","text":"In a deterministic benchmark, pre-failure work equals the total affected dependency closure size minus one, confirmed across a 96-configuration sweep with zero exceptions. A baseline executor that only detects failures after acting wasted up to nine steps on a doomed plan, while a validity-aware executor that checks dependency status before acting produced zero pre-failure steps in every tested case. Anyone debugging agents that follow broken plans can find deeper context on tracking state validity via daily.dev."}},{"@type":"Question","name":"Does checking whether a fact is still valid before acting always help an AI agent?","acceptedAnswer":{"@type":"Answer","text":"No, verification pays off only when the underlying fact actually changed. When a suspicious signal turned out to be a false alarm, the validity-aware executor spent one extra step confirming something that was fine (4 steps versus 3 for a baseline that never checked), while on a genuine change checking saved a step (6 versus 7) by avoiding a doomed action entirely. daily.dev surfaces this kind of trade-off analysis for engineers weighing verification costs in agent design."}},{"@type":"Question","name":"Can wasted steps from stale context cause an otherwise solvable AI agent task to fail entirely?","acceptedAnswer":{"@type":"Answer","text":"Yes, when the step budget is tight. In a swept-budget test on a value-change scenario, a baseline executor failed at budgets of 6, 7, and 8 steps because it wasted two steps on a doomed plan before recovering, while a validity-aware executor completed the same task at those same budgets since it avoided the wasted steps. daily.dev helps engineers keep tabs on findings like this when designing resource-constrained agent workflows."}}]}
```

