<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l" -->

---
title: Distributed, high-performance blob storage on SQLite
description: Zerodha built Blobasaur, a sharded, distributed blob storage server written in Rust on top of SQLite, to replace the unmaintained SSDB. It speaks the Redis and...
canonical: https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Distributed, high-performance blob storage on SQLite | daily.dev
og:description: Zerodha built Blobasaur, a sharded, distributed blob storage server written in Rust on top of SQLite, to replace the unmaintained SSDB. It speaks the Redis and...
og:url: https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l
og:image: https://api.daily.dev/og/posts/0zxxzfd9l.png
og:image:alt: Distributed, high-performance blob storage on SQLite
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Distributed, high-performance blob storage on SQLite

**[Hacker News](https://daily.dev/sources/hn)** · 14 min read · 3 upvotes · 1 comments

## Summary

Zerodha built Blobasaur, a sharded, distributed blob storage server written in Rust on top of SQLite, to replace the unmaintained SSDB. It speaks the Redis and Redis Cluster protocols, handling JSON blobs from kilobytes to megabytes at thousands of ops per second with single-digit millisecond latency. Benchmarks show 16 shards boost write throughput 3.3x over one shard, dropping median write latency from 8.2ms to 2.1ms. Zstd compression cuts production payloads roughly 11x on average. In production on a three-node cluster, p99 latency stays around 2ms. Blobasaur is open source under AGPLv3.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://zerodha.tech/blog/high-performance-distributed-blob-storage>

## Questions this post answers

### How much faster does sharding SQLite make write throughput compared to a single database file?

Splitting SQLite into 16 shards increased HSET write throughput 3.3 times compared to a single shard, going from 7,740 ops/s to 25,324 ops/s, while median write latency dropped from 8.2 ms to 2.1 ms. Read throughput stayed nearly identical since SQLite's WAL mode already serves concurrent readers without contention.

_Anyone scaling a SQLite-backed service can compare sharding strategies and benchmarks like this on daily.dev._

### Why did Zerodha choose SQLite instead of S3, ScyllaDB, or a raw filesystem for blob storage?

SQLite was chosen because it's up to 35% faster than raw filesystem I/O for small blobs, provides built-in ACID transactions and crash recovery, and avoids the operational overhead of alternatives. ScyllaDB had lower throughput for large blobs, S3 had high per-request latency and costs for write-heavy traffic, and raw filesystem storage required building atomic writes and TTL indexing from scratch.

_Developers weighing storage engines for high-throughput blob workloads can track comparisons like this on daily.dev._

### Why is Blobasaur written in Rust instead of Go even though most of the backend stack is Go?

Go's standard SQLite driver requires CGo, which complicates static builds, pins OS threads during C calls under high concurrency, and adds garbage-collection pressure when allocating and compressing multi-megabyte buffers. Rust, using sqlx, rusqlite, and tokio, calls SQLite's C API directly with predictable memory usage and ships as a single static binary.

_Engineers deciding between Rust and Go for SQLite-heavy services can follow real trade-offs like this on daily.dev._

## Community take

How the wider developer community reacted, aggregated from 2 discussions and 39 comments across lobsters, hackernews (as of 2026-11-19).

**TL;DR:** Discussion is dominated by skepticism about using SQLite for this workload, with several commenters pointing to alternatives (RocksDB, rqlite, ClickHouse, Ceph, plain files) and raising unanswered questions about replication, fault tolerance, and consistency.

**Sentiment:** 15% positive · 35% mixed · 50% skeptical

**The case for**

- Some acknowledge SQLite can outperform filesystems when dealing with millions of small files.
- A few note sharding and Zstd compression are reasonable, pragmatic choices for the stated workload.

**The pushback**

- Several argue SQLite's transaction/fsync commit overhead makes it a poor fit for high-throughput blob storage compared to RocksDB or LMDB.
- Multiple commenters criticize the lack of detail on replication, fault tolerance, and behavior when a node goes down unexpectedly.
- Some see this as reinventing existing solutions (OpenStack Swift, rqlite, ClickHouse, Ceph) rather than solving a novel problem.
- One commenter suggests storing diffs instead of full blobs could have avoided much of the write load entirely.
- Criticism that basic SQLite behaviors (e.g., deleted rows not shrinking file size) are presented as surprising findings.

**By community**

- lobsters (mixed): No comments were provided, so the discussion tone can't be assessed beyond minimal engagement.
- hackernews (skeptical): Many developers question the choice of SQLite over alternatives and push back on missing details around replication and fault tolerance.

**Hottest debate:** Whether SQLite was the right foundation at all, versus alternatives like RocksDB, rqlite, ClickHouse, or plain files.

**Open questions**

- What is the consistency model and what happens if a node goes down unexpectedly?
- Is there real replication or backup support beyond basic filesystem copy tools?
- Why was SQLite chosen over alternatives like RocksDB given its transaction overhead?

**Highlights**

> I spent years trying to squeeze this use case out of sqlite. All roads lead to tears, it is not made for this use case. I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck. sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.
> — [anacrolix on hackernews · 2 comments](https://news.ycombinator.com/item?id=50559043)

> >  Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck. Why sqlite instead of RocksDB ? What you need at the end of the day for a blob storage is a transactional key value store. Which RocksDB will give you with much better throughput (and latency) and solves your transaction problem.
> — [adev\_ on hackernews](https://news.ycombinator.com/item?id=50561069)

> Sounds like it solves their particular problem, but hardly can be used by outsiders in the current state. Bad fault tolerance, no words on replication or backup. Offline rebalance... meh. I am not sure how I feel about their choice of Redis protocol.
> — [murkt on hackernews](https://news.ycombinator.com/item?id=50559004)

> What is the consistency model? What happens if a node is down unexpectedly?
> — [matja on hackernews](https://news.ycombinator.com/item?id=50558861)

> I wonder if they tried storing diffs instead, and compacting them once in a while. These are user preferences for UI setup, and they have a lot of write traffic. But probably each user changes their preferences only by a little bit each time. Their 1MB writes would go down to just the small diffs. They could store the diffs and transparently apply them before sending to the client. After some timeout or maybe after a certain number of diffs have accumulated, squash them all and write back. They’d still need a server and custom code to handle this but it would probably reduce the database load and SSD writes by a lot. And let them just use something off-the-shelf for storage.
> — [articulatepang on hackernews](https://news.ycombinator.com/item?id=50561040)

**Source threads**

- [lobsters](https://lobste.rs/s/bk5nfs/distributed_high_performance_blob) · 13 points · 0 comments
- [hackernews](https://news.ycombinator.com/item?id=50557537) · 150 points · 39 comments

## Community discussion

Top comments from developers on daily.dev.

**@amananand755** · 0 upvotes

> a cool and interesting innovation

## Similar posts on daily.dev

- [GitHub - russellromney/turbolite: SQLite VFS with sub-50ms cold queries from S3 \+ page-level compression and encryption](https://daily.dev/posts/github---russellromney-turbolite-sqlite-vfs-with-sub-50ms-cold-queries-from-s3-page-level-compres-9zseh6udt) · Hacker News · 13 upvotes · 1 comments
- [Home Made CobbleDB Replaces DynamoDB at Perplexity to Cut Query Latency 5x and Reduce Cloud Storage](https://daily.dev/posts/home-made-cobbledb-replaces-dynamodb-at-perplexity-to-cut-query-latency-5x-and-reduce-cloud-storage-ehapdrsri) · InfoQ · 1 upvotes · 0 comments

---

Tags: [#rust](https://daily.dev/tags/rust), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#sqlite](https://daily.dev/tags/sqlite)

[View this post on daily.dev](https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Distributed, high-performance blob storage on SQLite","url":"https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l"},"datePublished":"2026-11-19T11:34:40.017Z","dateModified":"2026-11-19T15:35:46.957Z","description":"Zerodha built Blobasaur, a sharded, distributed blob storage server written in Rust on top of SQLite, to replace the unmaintained SSDB. It speaks the Redis and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f2e5153cf001a7ab88c5a1f4853de2ad?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f2e5153cf001a7ab88c5a1f4853de2ad?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"rust,distributed-systems,sqlite","timeRequired":"PT14M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Distributed, high-performance blob storage on SQLite"}]}
{"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l","comment":[{"@type":"Comment","text":"a cool and interesting innovation","datePublished":"2026-11-20T13:56:04.156Z","url":"https://daily.dev/posts/0zxxzfd9l#c-S6mvFR3so","author":{"@type":"Person","name":"Aman Anand","url":"https://daily.dev/amananand755","image":"https://media.daily.dev/image/upload/s--IC7nsnTH--/f_auto,pg_1/v1794242586/avatars/avatar_HwTBgfjy0?_a=BAMAMiiw0"}}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/distributed-high-performance-blob-storage-on-sqlite-0zxxzfd9l#faq","mainEntity":[{"@type":"Question","name":"How much faster does sharding SQLite make write throughput compared to a single database file?","acceptedAnswer":{"@type":"Answer","text":"Splitting SQLite into 16 shards increased HSET write throughput 3.3 times compared to a single shard, going from 7,740 ops/s to 25,324 ops/s, while median write latency dropped from 8.2 ms to 2.1 ms. Read throughput stayed nearly identical since SQLite's WAL mode already serves concurrent readers without contention. Anyone scaling a SQLite-backed service can compare sharding strategies and benchmarks like this on daily.dev."}},{"@type":"Question","name":"Why did Zerodha choose SQLite instead of S3, ScyllaDB, or a raw filesystem for blob storage?","acceptedAnswer":{"@type":"Answer","text":"SQLite was chosen because it's up to 35% faster than raw filesystem I/O for small blobs, provides built-in ACID transactions and crash recovery, and avoids the operational overhead of alternatives. ScyllaDB had lower throughput for large blobs, S3 had high per-request latency and costs for write-heavy traffic, and raw filesystem storage required building atomic writes and TTL indexing from scratch. Developers weighing storage engines for high-throughput blob workloads can track comparisons like this on daily.dev."}},{"@type":"Question","name":"Why is Blobasaur written in Rust instead of Go even though most of the backend stack is Go?","acceptedAnswer":{"@type":"Answer","text":"Go's standard SQLite driver requires CGo, which complicates static builds, pins OS threads during C calls under high concurrency, and adds garbage-collection pressure when allocating and compressing multi-megabyte buffers. Rust, using sqlx, rusqlite, and tokio, calls SQLite's C API directly with predictable memory usage and ships as a single static binary. Engineers deciding between Rust and Go for SQLite-heavy services can follow real trade-offs like this on daily.dev."}}]}
```

