--- title: Rare Not Random: Using Token Efficiency for Secrets Scanning description: Token Efficiency is proposed as a superior alternative to Shannon Entropy for filtering candidate secrets in secrets scanning. By measuring the ratio of string... canonical: https://daily.dev/posts/rare-not-random-using-token-efficiency-for-secrets-scanning-owb4ti3lb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Rare Not Random: Using Token Efficiency for Secrets Scanning | daily.dev og:description: Token Efficiency is proposed as a superior alternative to Shannon Entropy for filtering candidate secrets in secrets scanning. By measuring the ratio of string... og:url: https://daily.dev/posts/rare-not-random-using-token-efficiency-for-secrets-scanning-owb4ti3lb og:image: https://api.daily.dev/og/posts/OwB4TI3lB.png og:image:alt: Rare Not Random: Using Token Efficiency for Secrets Scanning og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Rare Not Random: Using Token Efficiency for Secrets Scanning **[Aikido Security](https://daily.dev/sources/aikido-security)** · 12 min read · 0 upvotes · 0 comments ## Summary Token Efficiency is proposed as a superior alternative to Shannon Entropy for filtering candidate secrets in secrets scanning. By measuring the ratio of string length to BPE token count using the cl100k_base tokenizer, rare or non-natural-language strings (like API keys and passwords) produce low token efficiency scores. Benchmarked against the CredData dataset, Token Efficiency combined with a word filter achieves an F1 of 0.874 vs Entropy+word filter's 0.715. Integrated into Betterleaks (a Gitleaks successor), the approach reaches an F1 of 0.892, beating CredSweeper's 0.85. The post includes implementation code, benchmark results, and concrete examples showing cases where entropy fails but token efficiency succeeds. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [How We Increased Shannon Entropy Throughput in Go by 41% Using a Lookup Table.](https://daily.dev/posts/how-we-increased-shannon-entropy-throughput-in-go-by-41-using-a-lookup-table--qu11tyrhk) · Awesome Go · 1 upvotes · 0 comments - [Gitleaks creator returns with Betterleaks, an open source secrets scanner for the agentic era](https://daily.dev/posts/gitleaks-creator-returns-with-betterleaks-an-open-source-secrets-scanner-for-the-agentic-era-cnjlbdmza) · The New Stack · 0 upvotes · 0 comments - [Better generic secrets detection starts with finding non-secrets](https://daily.dev/posts/better-generic-secrets-detection-starts-with-finding-non-secrets-dlj9j3gdr) · Aikido Security · 0 upvotes · 0 comments - [Don’t Break the Agent: Lessons in Token Optimization](https://daily.dev/posts/don-t-break-the-agent-lessons-in-token-optimization-fgdkm8bwd) · JFrog · 1 upvotes · 1 comments --- Tags: [#security](https://daily.dev/tags/security) [View this post on daily.dev](https://daily.dev/posts/rare-not-random-using-token-efficiency-for-secrets-scanning-owb4ti3lb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Rare Not Random: Using Token Efficiency for Secrets Scanning","url":"https://daily.dev/posts/rare-not-random-using-token-efficiency-for-secrets-scanning-owb4ti3lb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/rare-not-random-using-token-efficiency-for-secrets-scanning-owb4ti3lb"},"datePublished":"2026-03-31T12:25:37.521Z","dateModified":"2026-03-31T12:39:28.073Z","description":"Token Efficiency is proposed as a superior alternative to Shannon Entropy for filtering candidate secrets in secrets scanning. By measuring the ratio of string...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d1c72cf30152c7cbb4f8ac36aca369f3?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d1c72cf30152c7cbb4f8ac36aca369f3?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Aikido Security","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Aikido Security","logo":"https://media.daily.dev/image/upload/s--vJDqDYv_--/f_auto,q_auto/v1774959928/logos/aikido-security?_a=BAMAMiWQ0","url":"https://daily.dev/sources/aikido-security"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/rare-not-random-using-token-efficiency-for-secrets-scanning-owb4ti3lb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"security","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Aikido Security","item":"https://daily.dev/sources/aikido-security"},{"@type":"ListItem","position":3,"name":"Rare Not Random: Using Token Efficiency for Secrets Scanning"}]} ```