<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi" -->

---
title: Processing 11 million rows in minutes instead of hours
description: A developer optimized their blog's analytics system to process 11 million events, improving performance from 30 to 14,000 events per second through systematic...
canonical: https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Processing 11 million rows in minutes instead of hours | daily.dev
og:description: A developer optimized their blog's analytics system to process 11 million events, improving performance from 30 to 14,000 events per second through systematic...
og:url: https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi
og:image: https://api.daily.dev/og/posts/JGaGDggzi.png
og:image:alt: Processing 11 million rows in minutes instead of hours
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Processing 11 million rows in minutes instead of hours

**[stitcher.io](https://daily.dev/sources/stitcher)** · 10 min read · 277 upvotes · 5 comments

## Summary

A developer optimized their blog's analytics system to process 11 million events, improving performance from 30 to 14,000 events per second through systematic changes: removing unnecessary database sorting, reversing nested loops, ditching the ORM for raw queries, replacing closures with while loops, fixing framework serialization overhead, and using indexed ID-based pagination instead of offsets. The rebuild time dropped from 50 hours to 10 minutes per projector.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://stitcher.io/blog/processing-11-million-rows>

## Community discussion

Top comments from developers on daily.dev.

**@fatah7** · 9 upvotes

> ```php
>  // Loop over each projector
> foreach ($projectors as $projector) {
>         // Loop over each event
>         foreach ($storedEvents as $storedEvent) {
>             // …
>         }
>     }
> ```
>
> Although i skim read this, but each time i see nested loop i always wondered is there a way to remove the other loop if each event can be processed to multiple projector why not using concurrency or parallelism (well it's php but still)? just a thought.

**@masiorama** · 2 upvotes

> Intersting article, I digged more into event sourcing after this. Thanks.

**@jack\_starborn** · 0 upvotes

> Impressive. Thx for sharing.

**@agustinbarrientos** · 0 upvotes

> Each projector would checkpoint the last committed event ID in my replay state so a late failure can't send ten million successful events through the loop again

## Similar posts on daily.dev

- [Once again processing 11 million rows, now in seconds](https://daily.dev/posts/once-again-processing-11-million-rows-now-in-seconds-oqf9cnfsn) · stitcher.io · 98 upvotes · 1 comments
- [What Replo learned optimizing 100\+ billion events in ClickHouse](https://daily.dev/posts/what-replo-learned-optimizing-100-billion-events-in-clickhouse-moguewyij) · ClickHouse · 2 upvotes · 0 comments
- [System Design: Your Database Is Fast. So Why Is Your API Taking 8 Seconds?](https://daily.dev/posts/system-design-your-database-is-fast-so-why-is-your-api-taking-8-seconds--8w3gkjckf) · Medium · 0 upvotes · 0 comments
- [Sometimes it's just better to load "all" the data](https://daily.dev/posts/sometimes-it-s-just-better-to-load-all-the-data-rzohxoihx) · We Are .NET · 65 upvotes · 2 comments

---

Tags: [#performance](https://daily.dev/tags/performance), [#database](https://daily.dev/tags/database), [#architecture](https://daily.dev/tags/architecture), [#php](https://daily.dev/tags/php)

[View this post on daily.dev](https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Processing 11 million rows in minutes instead of hours","url":"https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi"},"datePublished":"2026-01-19T14:16:09.599Z","dateModified":"2026-01-19T14:16:35.447Z","description":"A developer optimized their blog's analytics system to process 11 million events, improving performance from 30 to 14,000 events per second through systematic...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/63d53eeb4c6d538e1d039e4badf840d1?_a=AQAEulh","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/63d53eeb4c6d538e1d039e4badf840d1?_a=AQAEulh","isAccessibleForFree":true,"articleSection":"stitcher.io","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"stitcher.io","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/83a32fc89f9943f38951a2b360c0d3ec","url":"https://daily.dev/sources/stitcher"},"commentCount":5,"discussionUrl":"https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":277},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":5}],"keywords":"performance,database,architecture,php","timeRequired":"PT10M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"stitcher.io","item":"https://daily.dev/sources/stitcher"},{"@type":"ListItem","position":3,"name":"Processing 11 million rows in minutes instead of hours"}]}
{"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/processing-11-million-rows-in-minutes-instead-of-hours-jgagdggzi","comment":[{"@type":"Comment","text":"// Loop over each projector  \nforeach ($projectors as $projector) {\n        // Loop over each event\n        foreach ($storedEvents as $storedEvent) {\n            // …\n        }\n    }\n\nAlthough i skim read this, but each time i see nested loop i always wondered is there a way to remove the other loop if each event can be processed to multiple projector why not using concurrency or parallelism (well it’s php but still)? just a thought.","datePublished":"2026-01-20T04:01:18.311Z","url":"https://daily.dev/posts/JGaGDggzi#c-gUGew1Mzk","author":{"@type":"Person","name":"fatah","url":"https://daily.dev/fatah7","image":"https://lh3.googleusercontent.com/a/ACg8ocKgelgpJclmQWiuyZTLLAnFAgi3MhLgrQ9pVzqTdWDzCRc9dA=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9}},{"@type":"Comment","text":"Intersting article, I digged more into event sourcing after this. Thanks.","datePublished":"2026-01-21T10:15:45.157Z","url":"https://daily.dev/posts/JGaGDggzi#c-L9tlhAAYf","author":{"@type":"Person","name":"Marco Masi","url":"https://daily.dev/masiorama","image":"https://lh3.googleusercontent.com/a/AGNmyxaM-06_mkr2bqPCZQygmcrnzxUsivqXeEEi1ix6zQ=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2}},{"@type":"Comment","text":"Impressive. Thx for sharing.","datePublished":"2026-01-22T10:22:29.202Z","url":"https://daily.dev/posts/JGaGDggzi#c-gQYfihj2T","author":{"@type":"Person","name":"Jack Starborn","url":"https://daily.dev/jack_starborn","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"}},{"@type":"Comment","text":"Each projector would checkpoint the last committed event ID in my replay state so a late failure can’t send ten million successful events through the loop again","datePublished":"2026-08-30T14:45:59.048Z","url":"https://daily.dev/posts/JGaGDggzi#c-9JgkNKega","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"}}]}
```