--- title: 14× faster embeddings: how we rebuilt the ONNX path in... description: Manticore Search 27.1.5 ships a new ONNX Runtime backend for auto-embeddings that delivers ~14× faster throughput than the previous SentenceTransformers/Candle... canonical: https://daily.dev/posts/14-faster-embeddings-how-we-rebuilt-the-onnx-path-in-manticore-9ac8mmjgg twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: 14× faster embeddings: how we rebuilt the ONNX path in Manticore | daily.dev og:description: Manticore Search 27.1.5 ships a new ONNX Runtime backend for auto-embeddings that delivers ~14× faster throughput than the previous SentenceTransformers/Candle... og:url: https://daily.dev/posts/14-faster-embeddings-how-we-rebuilt-the-onnx-path-in-manticore-9ac8mmjgg og:image: https://api.daily.dev/og/posts/9AC8mMjGG.png og:image:alt: 14× faster embeddings: how we rebuilt the ONNX path in Manticore og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 14× faster embeddings: how we rebuilt the ONNX path in Manticore **[Manticore](https://daily.dev/sources/manticoresearch)** · 16 min read · 4 upvotes · 0 comments ## Summary Manticore Search 27.1.5 ships a new ONNX Runtime backend for auto-embeddings that delivers ~14× faster throughput than the previous SentenceTransformers/Candle path on CPU. The old path was stuck at 5–11 docs/sec regardless of concurrency or batch size; the new one reaches 70–233 docs/sec. Key engineering decisions: sharing a single ORT session across concurrent callers (safe on Linux/macOS per ORT's C API docs), processing one document per inference call instead of batching (padding overhead made batching slower with variable-length inputs), and disabling intra-op thread spinning to free CPU for the rest of the system. For maximum bulk ingest throughput, the recommended pattern is large batches (32–128 docs) from a single client thread, since ORT already parallelises internally. GPU support and Windows perf parity are planned for future releases. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much faster is the new ONNX Runtime embedding backend in Manticore Search 27.1.5 compared to the old Candle path? It is roughly 14x faster on average across the full threads x batch workload grid, on the same hardware, model, and weights. The old SentenceTransformers/Candle path stayed at 5-11 docs/sec regardless of configuration, while the new ONNX Runtime backend ranges from 70 to 233 docs/sec, with the advantage holding from 1 to 32 client threads. _Teams choosing an embedding backend for search infrastructure can track real-world benchmarks like this one on daily.dev._ ### Why does batching documents together make ONNX inference slower instead of faster in Manticore's embedding pipeline? Batching hurts because mixed-length text batches pad every row to the longest row's length, so the model does work proportional to batch_size times max_len times hidden_dim regardless of actual content, wasting cycles on padding tokens. Combined with ORT's intra-op thread pool spinning between dispatches and starving other work of CPU, one-document-per-call with spinning disabled outperformed batched inference in Manticore's tests. _Developers debugging unexpected inference slowdowns can compare notes on ONNX batching pitfalls via daily.dev._ ### How should I configure client threads and batch size to get maximum embedding throughput when bulk inserting into Manticore Search with ONNX auto-embeddings? Use a single client thread with a large batch size between 32 and 128 rather than many threads with small batches, since the ONNX backend already parallelizes internally. In benchmarks, 1 thread with batch size 64 hit 233 docs/sec, beating 8 threads at batch size 128 which reached only 147 docs/sec, because client-side fan-out just adds coordination overhead on top of ORT's own parallelism. _Engineers tuning bulk ingest throughput for vector search can weigh configuration tradeoffs like this on daily.dev._ ## Similar posts on daily.dev - [Faster KNN search in Manticore: 2-pass HNSW, batched distances, and AVX-512](https://daily.dev/posts/faster-knn-search-in-manticore-2-pass-hnsw-batched-distances-and-avx-512-w4vuktdoo) · Manticore · 0 upvotes · 0 comments - [Up to 580tps\! New Speed Record of Qwen3.5-397B-A17B on GPU for Agentic Workloads with TokenSpeed – PyTorch](https://daily.dev/posts/up-to-580tps-new-speed-record-of-qwen3-5-397b-a17b-on-gpu-for-agentic-workloads-with-tokenspeed-p-imoe4kosk) · PyTorch · 0 upvotes · 0 comments - [3x Faster Search: Parallel Test-Time Scaling with Instructed-Retriever-1](https://daily.dev/posts/3x-faster-search-parallel-test-time-scaling-with-instructed-retriever-1-mzfvguqym) · databricks · 0 upvotes · 0 comments --- Tags: [#rust](https://daily.dev/tags/rust), [#vector-search](https://daily.dev/tags/vector-search), [#embeddings](https://daily.dev/tags/embeddings), [#manticore-search](https://daily.dev/tags/manticore-search) [View this post on daily.dev](https://daily.dev/posts/14-faster-embeddings-how-we-rebuilt-the-onnx-path-in-manticore-9ac8mmjgg) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"14× faster embeddings: how we rebuilt the ONNX path in Manticore","url":"https://daily.dev/posts/14-faster-embeddings-how-we-rebuilt-the-onnx-path-in-manticore-9ac8mmjgg","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/14-faster-embeddings-how-we-rebuilt-the-onnx-path-in-manticore-9ac8mmjgg"},"datePublished":"2026-07-09T07:50:01.916Z","dateModified":"2026-09-14T06:23:56.937Z","description":"Manticore Search 27.1.5 ships a new ONNX Runtime backend for auto-embeddings that delivers ~14× faster throughput than the previous SentenceTransformers/Candle...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/df2083cad551b512db37943e6b386478?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/df2083cad551b512db37943e6b386478?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Manticore","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Manticore","logo":"https://media.daily.dev/image/upload/s--FP5Ea08h--/f_auto/v1754225081/logos/manticoresearch","url":"https://daily.dev/sources/manticoresearch"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/14-faster-embeddings-how-we-rebuilt-the-onnx-path-in-manticore-9ac8mmjgg","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"rust,vector-search,embeddings,manticore-search","timeRequired":"PT16M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Manticore","item":"https://daily.dev/sources/manticoresearch"},{"@type":"ListItem","position":3,"name":"14× faster embeddings: how we rebuilt the ONNX path in Manticore"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/14-faster-embeddings-how-we-rebuilt-the-onnx-path-in-manticore-9ac8mmjgg#faq","mainEntity":[{"@type":"Question","name":"How much faster is the new ONNX Runtime embedding backend in Manticore Search 27.1.5 compared to the old Candle path?","acceptedAnswer":{"@type":"Answer","text":"It is roughly 14x faster on average across the full threads x batch workload grid, on the same hardware, model, and weights. The old SentenceTransformers/Candle path stayed at 5-11 docs/sec regardless of configuration, while the new ONNX Runtime backend ranges from 70 to 233 docs/sec, with the advantage holding from 1 to 32 client threads. Teams choosing an embedding backend for search infrastructure can track real-world benchmarks like this one on daily.dev."}},{"@type":"Question","name":"Why does batching documents together make ONNX inference slower instead of faster in Manticore's embedding pipeline?","acceptedAnswer":{"@type":"Answer","text":"Batching hurts because mixed-length text batches pad every row to the longest row's length, so the model does work proportional to batch_size times max_len times hidden_dim regardless of actual content, wasting cycles on padding tokens. Combined with ORT's intra-op thread pool spinning between dispatches and starving other work of CPU, one-document-per-call with spinning disabled outperformed batched inference in Manticore's tests. Developers debugging unexpected inference slowdowns can compare notes on ONNX batching pitfalls via daily.dev."}},{"@type":"Question","name":"How should I configure client threads and batch size to get maximum embedding throughput when bulk inserting into Manticore Search with ONNX auto-embeddings?","acceptedAnswer":{"@type":"Answer","text":"Use a single client thread with a large batch size between 32 and 128 rather than many threads with small batches, since the ONNX backend already parallelizes internally. In benchmarks, 1 thread with batch size 64 hit 233 docs/sec, beating 8 threads at batch size 128 which reached only 147 docs/sec, because client-side fan-out just adds coordination overhead on top of ORT's own parallelism. Engineers tuning bulk ingest throughput for vector search can weigh configuration tradeoffs like this on daily.dev."}}]} ```