--- title: AI Models Might Not Need Tokens Anymore! #ai #meta #aimodel description: A new Meta research paper explores training small language models directly on raw bytes instead of tokens. Byte-level models need only 256 symbols versus Llama... canonical: https://daily.dev/posts/ai-models-might-not-need-tokens-anymore-ai-meta-aimodel-44154rx1u twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AI Models Might Not Need Tokens Anymore! #ai #meta #aimodel | daily.dev og:description: A new Meta research paper explores training small language models directly on raw bytes instead of tokens. Byte-level models need only 256 symbols versus Llama... og:url: https://daily.dev/posts/ai-models-might-not-need-tokens-anymore-ai-meta-aimodel-44154rx1u og:image: https://api.daily.dev/og/posts/44154rx1u.png og:image:alt: AI Models Might Not Need Tokens Anymore! #ai #meta #aimodel og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AI Models Might Not Need Tokens Anymore! #ai #meta #aimodel **[YouTube](https://daily.dev/sources/youtube)** · 2 min read · 1 upvotes · 0 comments ## Summary A new Meta research paper explores training small language models directly on raw bytes instead of tokens. Byte-level models need only 256 symbols versus Llama 3's 128,000 tokens, and historically underperformed token-based models, but struggled to use distillation since teacher and student models needed matching tokenizers. Researchers introduced an 'end of token' symbol to convert a token-based teacher's (Llama 3 8B) probability distribution into exact byte-level predictions, enabling distillation across tokenizer boundaries. In experiments training 1B-parameter student models on up to a trillion bytes, byte models started behind but eventually overtook token models, needing roughly a sixth of the training data to match performance and projected to score about four points higher on benchmarks if trained further - though this gain remains a projection, not an achieved result. Byte-level distillation also compresses storage of full probability distributions dramatically, using about a fifth of the space while retaining full distributions instead of just top predictions. Trade-off: byte models are slower at inference since sequences become four to five times longer. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Meta's byte-level distillation method work for training small language models without a tokenizer? It converts a token-based teacher model's predictions into exact byte-level predictions using a special 'end of token' symbol that captures probability mass lost during conversion. Researchers used Llama 3 8B as the teacher to train 1B-parameter byte-level student models on up to a trillion bytes, letting distillation work even though teacher and student use different symbol systems. _Developers tracking new approaches to efficient model training can follow research breakdowns like this on daily.dev._ ### Do byte-level language models perform better than token-based models like Llama 3? Byte-level student models trained by distilling from Llama 3 8B started behind token-based models early in training but eventually overtook them, reaching the same score using only about a sixth of the training data. Researchers project byte models could score around four points higher on benchmarks with extended training, though that gain remains unverified and projected rather than achieved. _Anyone weighing tokenizer-based versus byte-level architectures can keep up with findings like this through daily.dev._ ### What is the inference speed trade-off for byte-level language models compared to token-based ones? Byte-level models are slower at inference because the same piece of text turns into a sequence four to five times longer than it would be with tokens, since each byte is processed individually instead of grouped into larger token chunks. This makes byte models better suited to training scenarios with large compute budgets rather than latency-sensitive inference. _Teams evaluating inference cost versus training efficiency trade-offs can track this kind of research on daily.dev._ ## Similar posts on daily.dev - [Multi-token prediction technique triples LLM inference speed without auxiliary draft models](https://daily.dev/posts/multi-token-prediction-technique-triples-llm-inference-speed-without-auxiliary-draft-models-hxaajvily) · InfoWorld · 1 upvotes · 0 comments - [tokens too cheap to meter](https://daily.dev/posts/tokens-too-cheap-to-meter-wekdxgrid) · Lobsters · 0 upvotes · 0 comments - [AI Crash Course: Tokens, Prediction and Temperature](https://daily.dev/posts/ai-crash-course-tokens-prediction-and-temperature-tm854yntp) · Telerik · 1 upvotes · 0 comments - [What Is a Token in AI? An Explainer](https://daily.dev/posts/what-is-a-token-in-ai-an-explainer-a3xndppfv) · Couchbase · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#nlp](https://daily.dev/tags/nlp), [#llama](https://daily.dev/tags/llama) [View this post on daily.dev](https://daily.dev/posts/ai-models-might-not-need-tokens-anymore-ai-meta-aimodel-44154rx1u) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AI Models Might Not Need Tokens Anymore! #ai #meta #aimodel","url":"https://daily.dev/posts/ai-models-might-not-need-tokens-anymore-ai-meta-aimodel-44154rx1u","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/ai-models-might-not-need-tokens-anymore-ai-meta-aimodel-44154rx1u"},"datePublished":"2026-10-04T05:43:00.058Z","dateModified":"2026-10-04T05:43:20.751Z","description":"A new Meta research paper explores training small language models directly on raw bytes instead of tokens. Byte-level models need only 256 symbols versus Llama...","image":"https://i.ytimg.com/vi/vIkRLhHoxuo/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/vIkRLhHoxuo/sddefault.jpg","isAccessibleForFree":true,"articleSection":"YouTube","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"YouTube","logo":"https://media.daily.dev/image/upload/s--W5zMumpP--/f_auto/v1725350181/logos/youtube","url":"https://daily.dev/sources/youtube"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/ai-models-might-not-need-tokens-anymore-ai-meta-aimodel-44154rx1u","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,nlp,llama","timeRequired":"PT2M","video":{"@type":"VideoObject","name":"AI Models Might Not Need Tokens Anymore! #ai #meta #aimodel","description":"A new Meta research paper explores training small language models directly on raw bytes instead of tokens. Byte-level models need only 256 symbols versus Llama...","thumbnailUrl":"https://i.ytimg.com/vi/vIkRLhHoxuo/sddefault.jpg","uploadDate":"2026-10-04T05:43:00.058Z","duration":"PT2M","url":"https://api.daily.dev/r/44154rx1u","embedUrl":"https://www.youtube.com/embed/vIkRLhHoxuo"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"YouTube","item":"https://daily.dev/sources/youtube"},{"@type":"ListItem","position":3,"name":"AI Models Might Not Need Tokens Anymore! #ai #meta #aimodel"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/ai-models-might-not-need-tokens-anymore-ai-meta-aimodel-44154rx1u#faq","mainEntity":[{"@type":"Question","name":"How does Meta's byte-level distillation method work for training small language models without a tokenizer?","acceptedAnswer":{"@type":"Answer","text":"It converts a token-based teacher model's predictions into exact byte-level predictions using a special 'end of token' symbol that captures probability mass lost during conversion. Researchers used Llama 3 8B as the teacher to train 1B-parameter byte-level student models on up to a trillion bytes, letting distillation work even though teacher and student use different symbol systems. Developers tracking new approaches to efficient model training can follow research breakdowns like this on daily.dev."}},{"@type":"Question","name":"Do byte-level language models perform better than token-based models like Llama 3?","acceptedAnswer":{"@type":"Answer","text":"Byte-level student models trained by distilling from Llama 3 8B started behind token-based models early in training but eventually overtook them, reaching the same score using only about a sixth of the training data. Researchers project byte models could score around four points higher on benchmarks with extended training, though that gain remains unverified and projected rather than achieved. Anyone weighing tokenizer-based versus byte-level architectures can keep up with findings like this through daily.dev."}},{"@type":"Question","name":"What is the inference speed trade-off for byte-level language models compared to token-based ones?","acceptedAnswer":{"@type":"Answer","text":"Byte-level models are slower at inference because the same piece of text turns into a sequence four to five times longer than it would be with tokens, since each byte is processed individually instead of grouped into larger token chunks. This makes byte models better suited to training scenarios with large compute budgets rather than latency-sensitive inference. Teams evaluating inference cost versus training efficiency trade-offs can track this kind of research on daily.dev."}}]} ```