--- title: Bloom Filters in KurrentDB | daily.dev description: KurrentDB experiences a significant throughput drop when writing events to many new streams because each write requires an index lookup to determine the last... canonical: https://daily.dev/posts/bloom-filters-in-kurrentdb-o74uiw2sp twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Bloom Filters in KurrentDB | daily.dev og:description: KurrentDB experiences a significant throughput drop when writing events to many new streams because each write requires an index lookup to determine the last... og:url: https://daily.dev/posts/bloom-filters-in-kurrentdb-o74uiw2sp og:image: https://api.daily.dev/og/posts/O74uIw2sP.png og:image:alt: Bloom Filters in KurrentDB og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Bloom Filters in KurrentDB **[Event Store](https://daily.dev/sources/eventstore)** · 5 min read · 0 upvotes · 0 comments ## Summary KurrentDB experiences a significant throughput drop when writing events to many new streams because each write requires an index lookup to determine the last event number, and cache misses force expensive index file checks. A Bloom filter solves this by quickly identifying streams that definitely do not exist, bypassing the index entirely. With a 15MB Bloom filter, throughput is maintained at ~50k/s throughout a 10-million-stream write test. The post explains how Bloom filters work (probabilistic set membership using bit arrays and multiple hashes), covers graceful degradation when the filter saturates, and details the KurrentDB implementation: memory-mapped file access, periodic checkpointing, and data integrity via per-cache-line hashing with corruption recovery. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does write throughput drop when creating a large number of new streams in KurrentDB? Throughput drops because writing to a brand-new stream is always a cache miss, forcing KurrentDB to check its on-disk indexes to determine the last event number for that stream. As more index files accumulate (KurrentDB batches 1 million events before flushing to disk), each check becomes more expensive, dropping throughput from around 45k/s to about 15k/s at the 1 million stream mark. _Anyone diagnosing similar throughput cliffs in their own datastore can follow deep dives like this on daily.dev._ ### What configuration option controls the bloom filter size in KurrentDB and what is its default? The bloom filter used for stream existence checks is configured with the --stream-existence-filter-size option, specified in bytes, with a default of 256mb. It's recommended to set it to 1-2x the expected number of streams in the database; changing the setting rebuilds the filter, and setting it to 0 disables it entirely. _Developers tuning KurrentDB deployments can track configuration details like this via daily.dev._ ### How does KurrentDB detect and recover from bloom filter data corruption on disk? KurrentDB stores a hash of the preceding 60 bytes in the last 4 bytes of each cache line, so corruption from events like a power cut is detected on startup. When corruption is found, the affected cache line's bits are all set to 1, keeping the filter functional at a small efficacy cost; if more than 5% of the file is corrupted, the filter is rebuilt entirely. _Engineers researching resilient storage designs can follow implementation details like this on daily.dev._ ## Similar posts on daily.dev - [Bloom Filters Explained: The Probabilistic Data Structure Powering Instagram, Google, and High-Scale Systems](https://daily.dev/posts/bloom-filters-explained-the-probabilistic-data-structure-powering-instagram-google-and-high-scale-9sh61vl6v) · freeCodeCamp · 2 upvotes · 0 comments - [Bloom Filters: Theory, Engineering Trade‑offs, and Implementation in Go](https://daily.dev/posts/bloom-filters-theory-engineering-trade-offs-and-implementation-in-go-cp65iybjt) · InfoQ · 1 upvotes · 0 comments - [Bloom filters: the niche trick behind a 16× faster API](https://daily.dev/posts/bloom-filters-the-niche-trick-behind-a-16-faster-api-a6bxja3xt) · Programming Digest · 29 upvotes · 0 comments - [Bloom Filters in Postgres: The Index Type Most Developers Overlook](https://daily.dev/posts/bloom-filters-in-postgres-the-index-type-most-developers-overlook-t9ufonv8o) · Prisma Blog · 0 upvotes · 0 comments - [Two Bits Are Better Than One: making bloom filters 2x more accurate](https://daily.dev/posts/two-bits-are-better-than-one-making-bloom-filters-2x-more-accurate-afbqkwq5f) · Hacker News · 0 upvotes · 0 comments --- Tags: [#performance](https://daily.dev/tags/performance), [#architecture](https://daily.dev/tags/architecture) [View this post on daily.dev](https://daily.dev/posts/bloom-filters-in-kurrentdb-o74uiw2sp) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Bloom Filters in KurrentDB","url":"https://daily.dev/posts/bloom-filters-in-kurrentdb-o74uiw2sp","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/bloom-filters-in-kurrentdb-o74uiw2sp"},"datePublished":"2026-08-04T14:30:29.291Z","dateModified":"2026-09-13T20:44:08.532Z","description":"KurrentDB experiences a significant throughput drop when writing events to many new streams because each write requires an index lookup to determine the last...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0bf417345780400bf24fe09ed9e89272?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0bf417345780400bf24fe09ed9e89272?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Event Store","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Event Store","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/6e7940e0a29e4b3cbfef11831ec252c2","url":"https://daily.dev/sources/eventstore"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/bloom-filters-in-kurrentdb-o74uiw2sp","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"performance,architecture","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Event Store","item":"https://daily.dev/sources/eventstore"},{"@type":"ListItem","position":3,"name":"Bloom Filters in KurrentDB"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/bloom-filters-in-kurrentdb-o74uiw2sp#faq","mainEntity":[{"@type":"Question","name":"Why does write throughput drop when creating a large number of new streams in KurrentDB?","acceptedAnswer":{"@type":"Answer","text":"Throughput drops because writing to a brand-new stream is always a cache miss, forcing KurrentDB to check its on-disk indexes to determine the last event number for that stream. As more index files accumulate (KurrentDB batches 1 million events before flushing to disk), each check becomes more expensive, dropping throughput from around 45k/s to about 15k/s at the 1 million stream mark. Anyone diagnosing similar throughput cliffs in their own datastore can follow deep dives like this on daily.dev."}},{"@type":"Question","name":"What configuration option controls the bloom filter size in KurrentDB and what is its default?","acceptedAnswer":{"@type":"Answer","text":"The bloom filter used for stream existence checks is configured with the --stream-existence-filter-size option, specified in bytes, with a default of 256mb. It's recommended to set it to 1-2x the expected number of streams in the database; changing the setting rebuilds the filter, and setting it to 0 disables it entirely. Developers tuning KurrentDB deployments can track configuration details like this via daily.dev."}},{"@type":"Question","name":"How does KurrentDB detect and recover from bloom filter data corruption on disk?","acceptedAnswer":{"@type":"Answer","text":"KurrentDB stores a hash of the preceding 60 bytes in the last 4 bytes of each cache line, so corruption from events like a power cut is detected on startup. When corruption is found, the affected cache line's bits are all set to 1, keeping the filter functional at a small efficacy cost; if more than 5% of the file is corrupted, the filter is rebuilt entirely. Engineers researching resilient storage designs can follow implementation details like this on daily.dev."}}]} ```