--- title: How to search for long hashes and IDs with dict='keywords description: Manticore Search's default keywords dictionary truncates tokens to 42 bytes after normalization, which breaks searching long values like SHA-256 hashes,... canonical: https://daily.dev/posts/how-to-search-for-long-hashes-and-ids-with-dict-keywords-5cna8pqrd twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How to search for long hashes and IDs with dict='keywords | daily.dev og:description: Manticore Search's default keywords dictionary truncates tokens to 42 bytes after normalization, which breaks searching long values like SHA-256 hashes,... og:url: https://daily.dev/posts/how-to-search-for-long-hashes-and-ids-with-dict-keywords-5cna8pqrd og:image: https://api.daily.dev/og/posts/5CNa8pQrd.png og:image:alt: How to search for long hashes and IDs with dict='keywords og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to search for long hashes and IDs with dict='keywords **[Manticore](https://daily.dev/sources/manticoresearch)** · 10 min read · 0 upvotes · 0 comments ## Summary Manticore Search's default keywords dictionary truncates tokens to 42 bytes after normalization, which breaks searching long values like SHA-256 hashes, message IDs, and long email addresses since different IDs sharing the same first 42 bytes become indistinguishable. The dict='keywords_32k' option, available since Manticore Search 27.1.1 (27.1.5+ recommended for migrations), raises the maximum token length to 32768 bytes, skipping oversized tokens with a warning instead of truncating them. The guide covers schema choices (string attribute vs string attribute indexed), exact vs full-text matching semantics, prefix/infix search with min_infix_len, blend_chars for emails, CALL KEYWORDS for inspecting tokenization, migration steps for RT and plain tables, why dict='crc' doesn't help, current limitations (no CALL SUGGEST, no percolate tables, no snippet highlighting, no full-text REGEX), and a warning against indexing secrets like API keys and tokens. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does searching for a full SHA-256 hash in Manticore Search return the wrong document? The default dict='keywords' dictionary truncates tokens to 42 bytes after normalization, both when indexing and when querying. Since a SHA-256 hash is 64 characters, two different hashes sharing the same first 42 bytes become indistinguishable, and a search on the full value can match the wrong document because the query itself gets truncated too. _Track dictionary and indexing changes like this on daily.dev before they cause silent search bugs in production._ ### How do I search for long hashes or IDs longer than 42 bytes in Manticore Search? Use dict='keywords_32k', available starting with Manticore Search 27.1.1 (27.1.5 or newer recommended for migrating existing tables), which raises the maximum normalized token length to 32768 bytes instead of the default 42. Tokens exceeding the new limit are skipped with a warning rather than silently truncated, and prefix and infix wildcard search remain supported. _Developers evaluating search engine configs for log and ID data can follow releases like this on daily.dev._ ### Does converting a Manticore Search table to dict='keywords_32k' automatically fix existing indexed documents? No, changing dict on an RT table via ALTER TABLE only affects documents added or replaced afterward; existing documents keep their old truncated tokens until reindexed. For plain tables, the configuration must be updated and the table rebuilt entirely from its data source, so a table can temporarily contain both truncated and complete tokens for seemingly identical documents. _Anyone planning a dictionary migration can use daily.dev to stay ahead of gotchas like partial reindexing._ ## Similar posts on daily.dev - [Manticore Search 27.1.5: Authentication, sharded tables, conversational search and faster vector search](https://daily.dev/posts/manticore-search-27-1-5-authentication-sharded-tables-conversational-search-and-faster-vector-sea-vaeuj65j7) · Manticore · 2 upvotes · 0 comments - [Inline Stopwords, Exceptions, and Wordforms](https://daily.dev/posts/inline-stopwords-exceptions-and-wordforms-4n5sirdee) · Manticore · 0 upvotes · 0 comments - [Manticore Search 17.5.1](https://daily.dev/posts/manticore-search-17-5-1-kxwg1lxjz) · Manticore · 0 upvotes · 0 comments - [Manticore Search 28.6.6: UUID document IDs, ordered GROUP](https://daily.dev/posts/manticore-search-28-6-6-uuid-document-ids-ordered-group-xpfg7udv7) · Manticore · 3 upvotes · 0 comments - [Manticore Search 14.1.0: Force Bigrams and Bug Fixes](https://daily.dev/posts/manticore-search-14-1-0-force-bigrams-and-bug-fixes-ha1p5dya3) · Manticore · 0 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#nlp](https://daily.dev/tags/nlp), [#full-text-search](https://daily.dev/tags/full-text-search), [#manticore-search](https://daily.dev/tags/manticore-search) [View this post on daily.dev](https://daily.dev/posts/how-to-search-for-long-hashes-and-ids-with-dict-keywords-5cna8pqrd) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How to search for long hashes and IDs with dict='keywords","url":"https://daily.dev/posts/how-to-search-for-long-hashes-and-ids-with-dict-keywords-5cna8pqrd","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-search-for-long-hashes-and-ids-with-dict-keywords-5cna8pqrd"},"datePublished":"2026-08-30T05:16:47.556Z","dateModified":"2026-08-30T05:17:12.090Z","description":"Manticore Search's default keywords dictionary truncates tokens to 42 bytes after normalization, which breaks searching long values like SHA-256 hashes,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/455e94d421b4544e2ed6d75de68b2c23?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/455e94d421b4544e2ed6d75de68b2c23?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Manticore","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Manticore","logo":"https://media.daily.dev/image/upload/s--FP5Ea08h--/f_auto/v1754225081/logos/manticoresearch","url":"https://daily.dev/sources/manticoresearch"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-to-search-for-long-hashes-and-ids-with-dict-keywords-5cna8pqrd","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"backend,nlp,full-text-search,manticore-search","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Manticore","item":"https://daily.dev/sources/manticoresearch"},{"@type":"ListItem","position":3,"name":"How to search for long hashes and IDs with dict='keywords"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-to-search-for-long-hashes-and-ids-with-dict-keywords-5cna8pqrd#faq","mainEntity":[{"@type":"Question","name":"Why does searching for a full SHA-256 hash in Manticore Search return the wrong document?","acceptedAnswer":{"@type":"Answer","text":"The default dict='keywords' dictionary truncates tokens to 42 bytes after normalization, both when indexing and when querying. Since a SHA-256 hash is 64 characters, two different hashes sharing the same first 42 bytes become indistinguishable, and a search on the full value can match the wrong document because the query itself gets truncated too. Track dictionary and indexing changes like this on daily.dev before they cause silent search bugs in production."}},{"@type":"Question","name":"How do I search for long hashes or IDs longer than 42 bytes in Manticore Search?","acceptedAnswer":{"@type":"Answer","text":"Use dict='keywords_32k', available starting with Manticore Search 27.1.1 (27.1.5 or newer recommended for migrating existing tables), which raises the maximum normalized token length to 32768 bytes instead of the default 42. Tokens exceeding the new limit are skipped with a warning rather than silently truncated, and prefix and infix wildcard search remain supported. Developers evaluating search engine configs for log and ID data can follow releases like this on daily.dev."}},{"@type":"Question","name":"Does converting a Manticore Search table to dict='keywords_32k' automatically fix existing indexed documents?","acceptedAnswer":{"@type":"Answer","text":"No, changing dict on an RT table via ALTER TABLE only affects documents added or replaced afterward; existing documents keep their old truncated tokens until reindexed. For plain tables, the configuration must be updated and the table rebuilt entirely from its data source, so a table can temporarily contain both truncated and complete tokens for seemingly identical documents. Anyone planning a dictionary migration can use daily.dev to stay ahead of gotchas like partial reindexing."}}]} ```