--- title: Identity resolution without the nightly batch | daily.dev description: Nightly batch rebuilds of identity graphs in advertising platforms introduce hours of latency that silently corrupt attribution, frequency capping, and consent... canonical: https://daily.dev/posts/identity-resolution-without-the-nightly-batch-zkcnwmjds twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Identity resolution without the nightly batch | daily.dev og:description: Nightly batch rebuilds of identity graphs in advertising platforms introduce hours of latency that silently corrupt attribution, frequency capping, and consent... og:url: https://daily.dev/posts/identity-resolution-without-the-nightly-batch-zkcnwmjds og:image: https://api.daily.dev/og/posts/zkCnWMJdS.png og:image:alt: Identity resolution without the nightly batch og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Identity resolution without the nightly batch **[SingleStore](https://daily.dev/sources/singlestore)** · 6 min read · 1 upvotes · 0 comments ## Summary Nightly batch rebuilds of identity graphs in advertising platforms introduce hours of latency that silently corrupt attribution, frequency capping, and consent propagation. When a household's devices are only linked during a 2 AM rebuild, same-day conversions get misattributed, users receive duplicate ads across devices, and GDPR opt-outs remain active for up to 17 hours — all documented in the platform's own audit logs. The standard match-rate metric is circular and masks this staleness. As probabilistic edges grow and graphs scale to billions of nodes, batch windows stretch past business hours, compounding the problem. The proposed alternative is continuous, incremental resolution using data sharding in a distributed SQL engine, where incoming signals update only the relevant shard rather than recomputing the entire graph. This reduces edge resolution to seconds, enabling real-time attribution and near-instant consent propagation while keeping the lakehouse for offline analytical workloads. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why does frequency capping fail even when each device stays within its ad limit Frequency caps are tracked per device until an identity graph merges those devices into one household or person, which typically only happens during a nightly rebuild. A person can receive the same ad a dozen times across TV, phone, and laptop in one afternoon while each device individually stays within policy, because the platform has no way to know the devices belong together until the next rebuild. _Anyone debugging cross-device ad frequency issues can track identity graph architecture discussions on daily.dev._ ### why can match rate metrics for identity graphs look healthy even when the graph is stale Match rate is circular because it measures agreement against the same graph it is evaluating, so an edge that is missing today and won't be created until the next nightly rebuild never counts against the score since the denominator has no knowledge the edge is absent. This lets platforms with stale graphs report match rates identical to those with current data despite producing very different attribution results. _Teams evaluating identity resolution vendors can follow deeper dives on data quality metrics via daily.dev._ ### how does nightly batch identity graph processing create GDPR compliance risk When a consumer opts out or withdraws consent, a graph that only updates during a nightly batch rebuild keeps that profile active and targetable for hours afterward, and those hours of processing are timestamped in the platform's own logs. This conflicts with GDPR Article 7 on easy consent withdrawal, Article 21 on immediate effect of an objection, and Article 17 on the right to erasure, turning a legal deadline into a best-effort update that regulators can use as evidence. _Engineers building consent-aware data pipelines can track GDPR-relevant architecture patterns on daily.dev._ ## Similar posts on daily.dev - [Why ad models need fresh context at decision time](https://daily.dev/posts/why-ad-models-need-fresh-context-at-decision-time-iucauvpzj) · SingleStore · 0 upvotes · 0 comments - [The 200ms latency: A developer’s guide to real-time personalization](https://daily.dev/posts/the-200ms-latency-a-developer-s-guide-to-real-time-personalization-u7s5bwkwz) · InfoWorld · 0 upvotes · 0 comments - [From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline](https://daily.dev/posts/from-batch-to-micro-batch-streaming-lessons-learned-the-hard-way-in-a-delta-index-pipeline-wdc2knjc0) · InfoQ · 0 upvotes · 0 comments - [Snowflake: One Platform for First-Party and Partner Identity Resolution](https://daily.dev/posts/snowflake-one-platform-for-first-party-and-partner-identity-resolution-tpqypcpdo) · Snowflake Community · 0 upvotes · 0 comments - [Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale](https://daily.dev/posts/stop-answering-the-same-question-twice-interval-aware-caching-for-druid-at-netflix-scale-qpsbylyw5) · Netflix TechBlog · 6 upvotes · 0 comments --- Tags: [#singlestore](https://daily.dev/tags/singlestore) [View this post on daily.dev](https://daily.dev/posts/identity-resolution-without-the-nightly-batch-zkcnwmjds) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Identity resolution without the nightly batch","url":"https://daily.dev/posts/identity-resolution-without-the-nightly-batch-zkcnwmjds","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/identity-resolution-without-the-nightly-batch-zkcnwmjds"},"datePublished":"2026-08-03T14:07:39.452Z","dateModified":"2026-09-14T08:04:53.600Z","description":"Nightly batch rebuilds of identity graphs in advertising platforms introduce hours of latency that silently corrupt attribution, frequency capping, and consent...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/feb5ba8eff1af56b5af628e2d0e51c80?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/feb5ba8eff1af56b5af628e2d0e51c80?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"SingleStore","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"SingleStore","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/e2118b72d715435c84d6060bbe11d275","url":"https://daily.dev/sources/singlestore"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/identity-resolution-without-the-nightly-batch-zkcnwmjds","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"singlestore","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"SingleStore","item":"https://daily.dev/sources/singlestore"},{"@type":"ListItem","position":3,"name":"Identity resolution without the nightly batch"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/identity-resolution-without-the-nightly-batch-zkcnwmjds#faq","mainEntity":[{"@type":"Question","name":"why does frequency capping fail even when each device stays within its ad limit","acceptedAnswer":{"@type":"Answer","text":"Frequency caps are tracked per device until an identity graph merges those devices into one household or person, which typically only happens during a nightly rebuild. A person can receive the same ad a dozen times across TV, phone, and laptop in one afternoon while each device individually stays within policy, because the platform has no way to know the devices belong together until the next rebuild. Anyone debugging cross-device ad frequency issues can track identity graph architecture discussions on daily.dev."}},{"@type":"Question","name":"why can match rate metrics for identity graphs look healthy even when the graph is stale","acceptedAnswer":{"@type":"Answer","text":"Match rate is circular because it measures agreement against the same graph it is evaluating, so an edge that is missing today and won't be created until the next nightly rebuild never counts against the score since the denominator has no knowledge the edge is absent. This lets platforms with stale graphs report match rates identical to those with current data despite producing very different attribution results. Teams evaluating identity resolution vendors can follow deeper dives on data quality metrics via daily.dev."}},{"@type":"Question","name":"how does nightly batch identity graph processing create GDPR compliance risk","acceptedAnswer":{"@type":"Answer","text":"When a consumer opts out or withdraws consent, a graph that only updates during a nightly batch rebuild keeps that profile active and targetable for hours afterward, and those hours of processing are timestamped in the platform's own logs. This conflicts with GDPR Article 7 on easy consent withdrawal, Article 21 on immediate effect of an objection, and Article 17 on the right to erasure, turning a legal deadline into a best-effort update that regulators can use as evidence. Engineers building consent-aware data pipelines can track GDPR-relevant architecture patterns on daily.dev."}}]} ```