--- title: Designing a Point-in-Time Backtest You Can Actually Trust description: Building a trustworthy backtest for predictive ML models requires strict point-in-time discipline. Five rules are outlined: (1) only use features from data... canonical: https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Designing a Point-in-Time Backtest You Can Actually Trust | daily.dev og:description: Building a trustworthy backtest for predictive ML models requires strict point-in-time discipline. Five rules are outlined: (1) only use features from data... og:url: https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj og:image: https://api.daily.dev/og/posts/AE4li3wYj.png og:image:alt: Designing a Point-in-Time Backtest You Can Actually Trust og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Designing a Point-in-Time Backtest You Can Actually Trust **[Medium](https://daily.dev/sources/medium_js)** · 9 min read · 3 upvotes · 0 comments ## Summary Building a trustworthy backtest for predictive ML models requires strict point-in-time discipline. Five rules are outlined: (1) only use features from data dated at or before the scoring date T, (2) use time-based folds instead of random cross-validation, (3) measure lift over base rate rather than accuracy for rare events, (4) ensure the label is derived from a source independent of the features, and (5) pre-register success criteria before seeing results. A real-estate teardown prediction model illustrates the stakes: a seemingly impressive 5.3x lift collapsed to 0x when the label was found to be a circular function of the feature, and the final shipped model stands on an honest 2x signal from independent permit data. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I avoid data leakage when backtesting a predictive model on rare historical events? Build every feature from data dated at or before the scoring date, split train and test sets by time rather than randomly, and validate against a label derived from a source independent of the features. A teardown-prediction model's headline 5.3x lift collapsed to no signal when tested against an independent permit-based label instead of a circular assessment-value proxy, revealing the real signal was closer to 2x. _Teams shipping rare-event models track backtesting pitfalls like data leakage and circular labels on daily.dev._ ### Why is accuracy a misleading metric for a rare-event prediction model? A model that always predicts the negative class scores over 99% accuracy on an event occurring in under 1% of cases, despite having zero predictive value. Instead, lead with the base rate and measure lift (how much richer the top-ranked slice is versus chance), alongside precision-at-k, capture at k, and area under the precision-recall curve, bootstrapping confidence intervals since positives are scarce. _Anyone evaluating rare-event models on daily.dev can weigh lift versus accuracy before trusting a backtest._ ### What happens when a validation label is derived from the same data as the predictive feature? The backtest measures arithmetic rather than prediction, producing a falsely inflated result. In one case, a label built from a building-value spike (a Proposition 13 assessment artifact) was scored against a feature that was the land-to-improvement value ratio; since both derive from the same underlying number, the reported 5.3x lift was a mirror, not real signal, and required an independent permit-based label to expose. _Developers building label pipelines for ML models can review circular-label pitfalls like this via daily.dev._ ## Similar posts on daily.dev - [The 10 Golden Rules of Time Series Forecasting](https://daily.dev/posts/the-10-golden-rules-of-time-series-forecasting-q9eq2qbd8) · R-bloggers · 0 upvotes · 0 comments - [Why Powerful Machine Learning Is Deceptively Easy](https://daily.dev/posts/why-powerful-machine-learning-is-deceptively-easy-rgerlx9mc) · Towards Data Science · 3 upvotes · 0 comments - [Getting Back to Basics](https://daily.dev/posts/getting-back-to-basics-8dse6eor7) · Simple Thread · 75 upvotes · 2 comments - [Feature Engineering in 2025: Why Your Model Isn’t the Problem \(Your Data Is\)](https://daily.dev/posts/feature-engineering-in-2025-why-your-model-isn-t-the-problem-your-data-is--6q5qometb) · Medium · 0 upvotes · 0 comments - [How to Actually Use Train, Validation, and Test Sets in ML](https://daily.dev/posts/how-to-actually-use-train-validation-and-test-sets-in-ml-osiynng9m) · Daily Dose of Data Science \| Avi Chawla \| Substack · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#data-science](https://daily.dev/tags/data-science) [View this post on daily.dev](https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Designing a Point-in-Time Backtest You Can Actually Trust","url":"https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj"},"datePublished":"2026-07-23T22:25:34.574Z","dateModified":"2026-09-13T21:54:43.978Z","description":"Building a trustworthy backtest for predictive ML models requires strict point-in-time discipline. Five rules are outlined: (1) only use features from data...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/df04513efe30d512b00fa803b492bbd8?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/df04513efe30d512b00fa803b492bbd8?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,data-science","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Designing a Point-in-Time Backtest You Can Actually Trust"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/designing-a-point-in-time-backtest-you-can-actually-trust-ae4li3wyj#faq","mainEntity":[{"@type":"Question","name":"How do I avoid data leakage when backtesting a predictive model on rare historical events?","acceptedAnswer":{"@type":"Answer","text":"Build every feature from data dated at or before the scoring date, split train and test sets by time rather than randomly, and validate against a label derived from a source independent of the features. A teardown-prediction model's headline 5.3x lift collapsed to no signal when tested against an independent permit-based label instead of a circular assessment-value proxy, revealing the real signal was closer to 2x. Teams shipping rare-event models track backtesting pitfalls like data leakage and circular labels on daily.dev."}},{"@type":"Question","name":"Why is accuracy a misleading metric for a rare-event prediction model?","acceptedAnswer":{"@type":"Answer","text":"A model that always predicts the negative class scores over 99% accuracy on an event occurring in under 1% of cases, despite having zero predictive value. Instead, lead with the base rate and measure lift (how much richer the top-ranked slice is versus chance), alongside precision-at-k, capture at k, and area under the precision-recall curve, bootstrapping confidence intervals since positives are scarce. Anyone evaluating rare-event models on daily.dev can weigh lift versus accuracy before trusting a backtest."}},{"@type":"Question","name":"What happens when a validation label is derived from the same data as the predictive feature?","acceptedAnswer":{"@type":"Answer","text":"The backtest measures arithmetic rather than prediction, producing a falsely inflated result. In one case, a label built from a building-value spike (a Proposition 13 assessment artifact) was scored against a feature that was the land-to-improvement value ratio; since both derive from the same underlying number, the reported 5.3x lift was a mirror, not real signal, and required an independent permit-based label to expose. Developers building label pipelines for ML models can review circular-label pitfalls like this via daily.dev."}}]} ```