--- title: CSIRO Uses Serverless Architecture to Query 100x More... description: CSIRO built Serverless Beacon (sBeacon), a serverless architecture for querying genomic variant data without copying VCF files into a central database. It... canonical: https://daily.dev/posts/csiro-uses-serverless-architecture-to-query-100x-more-genomic-variants-on-aws-ybgijqhik twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: CSIRO Uses Serverless Architecture to Query 100x More Genomic Variants on AWS | daily.dev og:description: CSIRO built Serverless Beacon (sBeacon), a serverless architecture for querying genomic variant data without copying VCF files into a central database. It... og:url: https://daily.dev/posts/csiro-uses-serverless-architecture-to-query-100x-more-genomic-variants-on-aws-ybgijqhik og:image: https://api.daily.dev/og/posts/ybGiJqhiK.png og:image:alt: CSIRO Uses Serverless Architecture to Query 100x More Genomic Variants on AWS og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # CSIRO Uses Serverless Architecture to Query 100x More Genomic Variants on AWS **[InfoQ](https://daily.dev/sources/infoq)** · 3 min read · 0 upvotes · 0 comments ## Summary CSIRO built Serverless Beacon (sBeacon), a serverless architecture for querying genomic variant data without copying VCF files into a central database. It combines Amazon S3, Lambda, Athena, and DynamoDB, using bcftools and htslib to fetch only the needed genomic regions via HTTP byte range requests. Tested on chromosome 1 from the 1000 Genomes Project (2,504 samples, 1.1 GB compressed VCF), onboarding took 18 seconds and a 10,000 base pair query averaged 1.52 seconds at an estimated cost of USD 0.00013. Latency stayed consistent as matching variants grew from four to 400. Earlier sBeacon research in Nature Biotechnology reported support for up to 40 million individuals and 3 billion genomic locations. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I query genomic variant data stored as VCF files in S3 without loading it into a database? CSIRO's Serverless Beacon architecture uses AWS Lambda functions with bcftools and htslib to make HTTP byte range requests directly against compressed VCF files in S3, retrieving only the needed genomic region. Metadata like disease status and location is indexed separately in DynamoDB and Athena, avoiding a full data copy into a central database. _Teams designing cloud-native genomic query systems can track serverless architecture patterns like this on daily.dev._ ### What latency and cost can I expect from a serverless genomic variant query on AWS Lambda? In CSIRO's tests on chromosome 1 of the 1000 Genomes Project (2,504 samples, 1.1 GB compressed VCF), a query covering 10,000 base pairs averaged 1.52 seconds at an estimated compute cost of $0.00013. Latency stayed consistent as the number of matching variants rose from four to 400, and onboarding the dataset took 18 seconds. _Engineers estimating cloud costs for data-heavy query workloads can follow benchmarks like this on daily.dev._ ## Similar posts on daily.dev - [Amazon S3 Vectors now generally available with increased scale and performance](https://daily.dev/posts/amazon-s3-vectors-now-generally-available-with-increased-scale-and-performance-0gpqkus8w) · AWS · 0 upvotes · 0 comments - [Amazon S3 Vectors Reaches GA, Introducing "Storage-First" Architecture for RAG](https://daily.dev/posts/amazon-s3-vectors-reaches-ga-introducing-storage-first-architecture-for-rag-ebcqk2lpm) · InfoQ · 0 upvotes · 0 comments - [Decoupled by Design: Billion-Scale Vector Search](https://daily.dev/posts/decoupled-by-design-billion-scale-vector-search-puyxn3eah) · databricks · 26 upvotes · 0 comments - [Cloud Detection at Scale on a Laptop](https://daily.dev/posts/cloud-detection-at-scale-on-a-laptop-vlgt1lhyx) · ITNEXT · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#serverless](https://daily.dev/tags/serverless), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#aws-lambda](https://daily.dev/tags/aws-lambda) [View this post on daily.dev](https://daily.dev/posts/csiro-uses-serverless-architecture-to-query-100x-more-genomic-variants-on-aws-ybgijqhik) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"CSIRO Uses Serverless Architecture to Query 100x More Genomic Variants on AWS","url":"https://daily.dev/posts/csiro-uses-serverless-architecture-to-query-100x-more-genomic-variants-on-aws-ybgijqhik","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/csiro-uses-serverless-architecture-to-query-100x-more-genomic-variants-on-aws-ybgijqhik"},"datePublished":"2026-10-23T14:46:26.182Z","dateModified":"2026-10-23T14:46:55.447Z","description":"CSIRO built Serverless Beacon (sBeacon), a serverless architecture for querying genomic variant data without copying VCF files into a central database. It...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2f6bad70bb60c7ba49ee090a834185c4?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2f6bad70bb60c7ba49ee090a834185c4?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"InfoQ","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"InfoQ","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/afc3bced3e1e4b188dd9127017a60e0c","url":"https://daily.dev/sources/infoq"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/csiro-uses-serverless-architecture-to-query-100x-more-genomic-variants-on-aws-ybgijqhik","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,serverless,distributed-systems,aws-lambda","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"InfoQ","item":"https://daily.dev/sources/infoq"},{"@type":"ListItem","position":3,"name":"CSIRO Uses Serverless Architecture to Query 100x More Genomic Variants on AWS"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/csiro-uses-serverless-architecture-to-query-100x-more-genomic-variants-on-aws-ybgijqhik#faq","mainEntity":[{"@type":"Question","name":"How can I query genomic variant data stored as VCF files in S3 without loading it into a database?","acceptedAnswer":{"@type":"Answer","text":"CSIRO's Serverless Beacon architecture uses AWS Lambda functions with bcftools and htslib to make HTTP byte range requests directly against compressed VCF files in S3, retrieving only the needed genomic region. Metadata like disease status and location is indexed separately in DynamoDB and Athena, avoiding a full data copy into a central database. Teams designing cloud-native genomic query systems can track serverless architecture patterns like this on daily.dev."}},{"@type":"Question","name":"What latency and cost can I expect from a serverless genomic variant query on AWS Lambda?","acceptedAnswer":{"@type":"Answer","text":"In CSIRO's tests on chromosome 1 of the 1000 Genomes Project (2,504 samples, 1.1 GB compressed VCF), a query covering 10,000 base pairs averaged 1.52 seconds at an estimated compute cost of $0.00013. Latency stayed consistent as the number of matching variants rose from four to 400, and onboarding the dataset took 18 seconds. Engineers estimating cloud costs for data-heavy query workloads can follow benchmarks like this on daily.dev."}}]} ```