--- title: Tech in Five - Tokens & Tokenomics | daily.dev description: An explainer breaks down what tokens are in generative AI, showing how text is split into tokens and mapped to token IDs that models process mathematically. It... canonical: https://daily.dev/posts/tech-in-five---tokens-tokenomics-w3o04ubyy twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Tech in Five - Tokens & Tokenomics | daily.dev og:description: An explainer breaks down what tokens are in generative AI, showing how text is split into tokens and mapped to token IDs that models process mathematically. It... og:url: https://daily.dev/posts/tech-in-five---tokens-tokenomics-w3o04ubyy og:image: https://api.daily.dev/og/posts/W3O04UByy.png og:image:alt: Tech in Five - Tokens & Tokenomics og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Tech in Five - Tokens & Tokenomics **[John Savill's Technical Training](https://daily.dev/sources/john-savill)** · 7 min read · 2 upvotes · 0 comments ## Summary An explainer breaks down what tokens are in generative AI, showing how text is split into tokens and mapped to token IDs that models process mathematically. It covers how tokenization efficiency varies by language (English vs Hindi vs French) and by code, explains context windows, reasoning tokens, and how reasoning effort settings affect latency and cost. It closes with 'tokenomics' - the idea that choosing the cheapest model or lowest reasoning setting doesn't always minimize overall cost, since a capable model may solve a task in one pass while a cheaper model needs many expensive attempts. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does the same sentence use more tokens in Hindi than in English? Tokenization efficiency varies by language because a model's vocabulary, training data, and the language's own structure determine how text is split into tokens. In one example, an English sentence used nine tokens while the same sentence in Hindi used 23 tokens and in French used 15 tokens, since the tokenizer represents non-English text with more individual pieces, sometimes called a language tax. _Anyone budgeting AI costs across languages can follow token efficiency discussions on daily.dev._ ### What are reasoning tokens in AI models and why do they add cost? Reasoning tokens are internal tokens used for a model's planning and thought process before producing its final output; they are not part of the visible answer. Some models let users choose a reasoning effort level (low or high), and higher reasoning effort adds latency and consumes more tokens, increasing cost, so using high reasoning for simple questions can waste resources. _Developers weighing model reasoning settings against cost can track these tradeoffs on daily.dev._ ### Does picking the cheapest AI model with the lowest reasoning setting always minimize total cost? No, choosing the cheapest model with the lowest reasoning setting does not automatically produce the lowest overall cost. A capable medium-sized model with medium reasoning may answer correctly on the first attempt, while a cheaper model might require many more reasoning steps and repeated attempts, ultimately costing more overall. _Teams comparing model tiers for cost efficiency can follow tokenomics tradeoffs on daily.dev._ ## Similar posts on daily.dev - [What Is a Token in AI? An Explainer](https://daily.dev/posts/what-is-a-token-in-ai-an-explainer-a3xndppfv) · Couchbase · 0 upvotes · 0 comments - [The Fundamentals of AI: What every curious person should know about how language models work](https://daily.dev/posts/the-fundamentals-of-ai-what-every-curious-person-should-know-about-how-language-models-work-nfjhvcwso) · Cisco · 0 upvotes · 0 comments - [Tokenomics for FinOps Practitioners: The New Cost Frontier](https://daily.dev/posts/tokenomics-for-finops-practitioners-the-new-cost-frontier-vpqfgbetu) · finout · 0 upvotes · 0 comments - [The Real Token Economy Is Not About Spending Less. It Is About Thinking Smaller.](https://daily.dev/posts/the-real-token-economy-is-not-about-spending-less-it-is-about-thinking-smaller--nmhc4y3yc) · DEV · 6 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#nlp](https://daily.dev/tags/nlp) [View this post on daily.dev](https://daily.dev/posts/tech-in-five---tokens-tokenomics-w3o04ubyy) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Tech in Five - Tokens & Tokenomics","url":"https://daily.dev/posts/tech-in-five---tokens-tokenomics-w3o04ubyy","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/tech-in-five---tokens-tokenomics-w3o04ubyy"},"datePublished":"2026-08-13T14:27:51.751Z","dateModified":"2026-09-13T19:44:17.449Z","description":"An explainer breaks down what tokens are in generative AI, showing how text is split into tokens and mapped to token IDs that models process mathematically. It...","image":"https://i.ytimg.com/vi/PbngY1n4Awc/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/PbngY1n4Awc/sddefault.jpg","isAccessibleForFree":true,"articleSection":"John Savill's Technical Training","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"John Savill's Technical Training","logo":"https://media.daily.dev/image/upload/s--4rApg0fT--/f_auto,q_auto/v1780213220/logos/john-savill?_a=BAMAMiWQ0","url":"https://daily.dev/sources/john-savill"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/tech-in-five---tokens-tokenomics-w3o04ubyy","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,nlp","timeRequired":"PT7M","video":{"@type":"VideoObject","name":"Tech in Five - Tokens & Tokenomics","description":"An explainer breaks down what tokens are in generative AI, showing how text is split into tokens and mapped to token IDs that models process mathematically. It...","thumbnailUrl":"https://i.ytimg.com/vi/PbngY1n4Awc/sddefault.jpg","uploadDate":"2026-08-13T14:27:51.751Z","duration":"PT7M","url":"https://api.daily.dev/r/W3O04UByy","embedUrl":"https://www.youtube.com/embed/PbngY1n4Awc"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"John Savill's Technical Training","item":"https://daily.dev/sources/john-savill"},{"@type":"ListItem","position":3,"name":"Tech in Five - Tokens & Tokenomics"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/tech-in-five---tokens-tokenomics-w3o04ubyy#faq","mainEntity":[{"@type":"Question","name":"Why does the same sentence use more tokens in Hindi than in English?","acceptedAnswer":{"@type":"Answer","text":"Tokenization efficiency varies by language because a model's vocabulary, training data, and the language's own structure determine how text is split into tokens. In one example, an English sentence used nine tokens while the same sentence in Hindi used 23 tokens and in French used 15 tokens, since the tokenizer represents non-English text with more individual pieces, sometimes called a language tax. Anyone budgeting AI costs across languages can follow token efficiency discussions on daily.dev."}},{"@type":"Question","name":"What are reasoning tokens in AI models and why do they add cost?","acceptedAnswer":{"@type":"Answer","text":"Reasoning tokens are internal tokens used for a model's planning and thought process before producing its final output; they are not part of the visible answer. Some models let users choose a reasoning effort level (low or high), and higher reasoning effort adds latency and consumes more tokens, increasing cost, so using high reasoning for simple questions can waste resources. Developers weighing model reasoning settings against cost can track these tradeoffs on daily.dev."}},{"@type":"Question","name":"Does picking the cheapest AI model with the lowest reasoning setting always minimize total cost?","acceptedAnswer":{"@type":"Answer","text":"No, choosing the cheapest model with the lowest reasoning setting does not automatically produce the lowest overall cost. A capable medium-sized model with medium reasoning may answer correctly on the first attempt, while a cheaper model might require many more reasoning steps and repeated attempts, ultimately costing more overall. Teams comparing model tiers for cost efficiency can follow tokenomics tradeoffs on daily.dev."}}]} ```