--- title: Prometheus vs VictoriaMetrics: Ingestion Got Cheaper.... description: A team migrated from Prometheus to VictoriaMetrics expecting lower costs, and infrastructure pain did briefly ease. But the real culprit—a checkout metric with... canonical: https://daily.dev/posts/prometheus-vs-victoriametrics-ingestion-got-cheaper-cardinality-still-sent-the-bill--96qjukpel twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Prometheus vs VictoriaMetrics: Ingestion Got Cheaper. Cardinality Still Sent the Bill. | daily.dev og:description: A team migrated from Prometheus to VictoriaMetrics expecting lower costs, and infrastructure pain did briefly ease. But the real culprit—a checkout metric with... og:url: https://daily.dev/posts/prometheus-vs-victoriametrics-ingestion-got-cheaper-cardinality-still-sent-the-bill--96qjukpel og:image: https://api.daily.dev/og/posts/96qJUkPel.png og:image:alt: Prometheus vs VictoriaMetrics: Ingestion Got Cheaper. Cardinality Still Sent the Bill. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Prometheus vs VictoriaMetrics: Ingestion Got Cheaper. Cardinality Still Sent the Bill. **[Medium](https://daily.dev/sources/medium_js)** · 4 min read · 0 upvotes · 0 comments ## Summary A team migrated from Prometheus to VictoriaMetrics expecting lower costs, and infrastructure pain did briefly ease. But the real culprit—a checkout metric with an unbounded customer_id label—kept generating new time series regardless of backend, since a more efficient store only delayed the cardinality blowup rather than fixing it. The team resolved it by removing customer identifiers from metrics, pushing that data into traces and logs instead, and instituting a label budget: every new label needs a known upper bound, estimated series growth before merge, and monitoring of active-series count as a product metric. They also shifted benchmarking from synthetic ingestion tests to replaying real incident queries, since worst-case query latency during outages mattered more than average throughput. VictoriaMetrics was kept for its storage and retention profile, but the underlying lesson was that backend migrations can't fix bad instrumentation design. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did my active time series count keep growing after migrating from Prometheus to VictoriaMetrics? A more efficient backend does not fix bad label design, it only delays the symptoms. If a metric label like customer_id is unbounded, every unique value creates a new time series regardless of which storage engine is used; VictoriaMetrics tolerated the growth more efficiently, so alerts arrived later while the underlying cardinality problem kept expanding. _Teams debugging cardinality blowups after a metrics migration can compare backend tradeoffs on daily.dev._ ### How do I stop unbounded labels from causing cardinality explosions in Prometheus-style metrics? Remove high-cardinality identifiers, such as customer IDs, from metric labels and push that investigation path into traces and logs instead, keeping metrics for bounded aggregate questions. Then enforce a review rule requiring every new label to have a known upper bound, an estimated series count before merge, and monitoring of active-series growth as its own product metric. _Engineers designing observability label schemas can track cardinality best practices on daily.dev._ ### Should I benchmark a metrics backend with synthetic ingestion tests or real production queries? Real production queries matter more, because a metrics backend is read by humans under pressure and the worst important query needs to return before an engineer gives up and opens logs instead. Replaying actual workloads like on-call dashboards, capacity reports, and alert rule aggregations revealed problems that synthetic ingestion benchmarks missed entirely. _Anyone evaluating observability tools for incident response can dig into benchmarking approaches on daily.dev._ ## Similar posts on daily.dev - [Airbnb Migrates High-Volume Metrics Pipeline to OpenTelemetry](https://daily.dev/posts/airbnb-migrates-high-volume-metrics-pipeline-to-opentelemetry-ytqavyfqp) · InfoQ · 1 upvotes · 0 comments - [The cost of knowledge](https://daily.dev/posts/the-cost-of-knowledge-yign5zfjs) · Coralogix · 0 upvotes · 0 comments - [High Cardinality Metrics: How Prometheus and ClickHouse Handle Scale](https://daily.dev/posts/high-cardinality-metrics-how-prometheus-and-clickhouse-handle-scale-o0heii5ky) · Last9 · 1 upvotes · 0 comments - [ClickHouse vs Prometheus for High Cardinality, Part 1: Understanding the Problem](https://daily.dev/posts/clickhouse-vs-prometheus-for-high-cardinality-part-1-understanding-the-problem-jfrzmntpc) · ClickHouse · 0 upvotes · 0 comments - [Infinite Cardinality Metrics: Custom metrics built for modern systems](https://daily.dev/posts/infinite-cardinality-metrics-custom-metrics-built-for-modern-systems-anxnytfhl) · Datadog · 0 upvotes · 0 comments --- Tags: [#observability](https://daily.dev/tags/observability), [#prometheus](https://daily.dev/tags/prometheus) [View this post on daily.dev](https://daily.dev/posts/prometheus-vs-victoriametrics-ingestion-got-cheaper-cardinality-still-sent-the-bill--96qjukpel) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Prometheus vs VictoriaMetrics: Ingestion Got Cheaper. Cardinality Still Sent the Bill.","url":"https://daily.dev/posts/prometheus-vs-victoriametrics-ingestion-got-cheaper-cardinality-still-sent-the-bill--96qjukpel","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/prometheus-vs-victoriametrics-ingestion-got-cheaper-cardinality-still-sent-the-bill--96qjukpel"},"datePublished":"2026-09-01T00:38:23.767Z","dateModified":"2026-09-01T00:51:59.498Z","description":"A team migrated from Prometheus to VictoriaMetrics expecting lower costs, and infrastructure pain did briefly ease. But the real culprit—a checkout metric with...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/128d173000a51f3b69d2571f280a2212?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/128d173000a51f3b69d2571f280a2212?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/prometheus-vs-victoriametrics-ingestion-got-cheaper-cardinality-still-sent-the-bill--96qjukpel","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"observability,prometheus","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Prometheus vs VictoriaMetrics: Ingestion Got Cheaper. Cardinality Still Sent the Bill."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/prometheus-vs-victoriametrics-ingestion-got-cheaper-cardinality-still-sent-the-bill--96qjukpel#faq","mainEntity":[{"@type":"Question","name":"Why did my active time series count keep growing after migrating from Prometheus to VictoriaMetrics?","acceptedAnswer":{"@type":"Answer","text":"A more efficient backend does not fix bad label design, it only delays the symptoms. If a metric label like customer_id is unbounded, every unique value creates a new time series regardless of which storage engine is used; VictoriaMetrics tolerated the growth more efficiently, so alerts arrived later while the underlying cardinality problem kept expanding. Teams debugging cardinality blowups after a metrics migration can compare backend tradeoffs on daily.dev."}},{"@type":"Question","name":"How do I stop unbounded labels from causing cardinality explosions in Prometheus-style metrics?","acceptedAnswer":{"@type":"Answer","text":"Remove high-cardinality identifiers, such as customer IDs, from metric labels and push that investigation path into traces and logs instead, keeping metrics for bounded aggregate questions. Then enforce a review rule requiring every new label to have a known upper bound, an estimated series count before merge, and monitoring of active-series growth as its own product metric. Engineers designing observability label schemas can track cardinality best practices on daily.dev."}},{"@type":"Question","name":"Should I benchmark a metrics backend with synthetic ingestion tests or real production queries?","acceptedAnswer":{"@type":"Answer","text":"Real production queries matter more, because a metrics backend is read by humans under pressure and the worst important query needs to return before an engineer gives up and opens logs instead. Replaying actual workloads like on-call dashboards, capacity reports, and alert rule aggregations revealed problems that synthetic ingestion benchmarks missed entirely. Anyone evaluating observability tools for incident response can dig into benchmarking approaches on daily.dev."}}]} ```