<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/OvkjarWnB" -->

---
title: Knowledge distillation during mid-training improves...
description: A study on knowledge distillation finds that the standard forward KL distillation objective behaves differently depending on training phase. During...
canonical: https://daily.dev/posts/knowledge-distillation-during-mid-training-improves-reasoning-but-slows-factual-recall-ovkjarwnb
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Knowledge distillation during mid-training improves reasoning but slows factual recall | daily.dev
og:description: A study on knowledge distillation finds that the standard forward KL distillation objective behaves differently depending on training phase. During...
og:url: https://daily.dev/posts/knowledge-distillation-during-mid-training-improves-reasoning-but-slows-factual-recall-ovkjarwnb
og:image: https://api.daily.dev/og/posts/OvkjarWnB.png
og:image:alt: Knowledge distillation during mid-training improves reasoning but slows factual recall
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Knowledge distillation during mid-training improves reasoning but slows factual recall

**[Collections](https://daily.dev/sources/collections)** · 1 min read · 9 upvotes · 0 comments

## Summary

A study on knowledge distillation finds that the standard forward KL distillation objective behaves differently depending on training phase. During pre-training it boosts both reasoning and factual recall, but during mid-training it improves reasoning while slowing factual recall acquisition. The researchers propose Switch Distillation, which routes between distillation loss and standard cross-entropy based on teacher confidence (predictive entropy), only mimicking the teacher on tokens where it is confident. This avoids forcing the student to copy the teacher on uncertain tokens, where factual knowledge transfer tends to break down.

## Content

A recent study on model distillation finds that forward Kullback-Leibler (KL) distillation - the standard formulation - behaves differently depending on when it's applied during training.

During pre-training, forward KL distillation improves both reasoning and factual recall compared to standard next-token prediction. But during mid-training, the picture splits: reasoning keeps improving while factual recall acquisition actually slows down. In other words, a distilled student model picks up how its teacher reasons rather than what it knows.

To address this, the researchers propose **Switch Distillation**: a mid-training objective that selectively applies distillation only on tokens where the teacher is confident, using teacher predictive entropy as a routing signal. When the teacher isn't confident, it falls back to standard cross-entropy loss. The idea is simple - don't force the student to mimic the teacher on tokens where the teacher itself is uncertain, since that's likely where factual knowledge transfer breaks down.

The practical implication is that the standard KD setup isn't a safe default across all training phases. If you're doing mid-training and care about factual recall, vanilla forward KL distillation may be quietly working against you even as your reasoning benchmarks look fine.

## Questions this post answers

### Does knowledge distillation during mid-training hurt factual recall in language models?

Yes, forward KL distillation applied during mid-training slows factual recall acquisition even though it continues to improve reasoning ability. This differs from pre-training, where forward KL distillation improves both reasoning and factual recall simultaneously. The effect means a distilled student model tends to absorb the teacher's reasoning patterns more than its factual knowledge during this phase.

_daily.dev surfaces research like this for engineers tuning distillation pipelines before scaling training runs._

### What is Switch Distillation and how does it fix knowledge distillation problems?

Switch Distillation is a mid-training objective that applies distillation loss only on tokens where the teacher model shows high predictive confidence, using teacher entropy as the routing signal. On tokens where the teacher is uncertain, it falls back to standard cross-entropy loss instead of forcing the student to imitate the teacher, since uncertain tokens are where factual knowledge transfer tends to break down.

_Teams weighing distillation strategies can track approaches like this on daily.dev before committing to a training recipe._

## Similar posts on daily.dev

- [​Implement Knowledge Distillation from Scratch​](https://daily.dev/posts/implement-knowledge-distillation-from-scratch--vieqsxirg) · Daily Dose of Data Science \| Avi Chawla \| Substack · 1 upvotes · 0 comments
- [How Big Models Teach Small Models to Be Smart](https://daily.dev/posts/how-big-models-teach-small-models-to-be-smart-3wi68af3j) · ByteByteGo · 8 upvotes · 0 comments

---

Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#data-science](https://daily.dev/tags/data-science)

[View this post on daily.dev](https://daily.dev/posts/knowledge-distillation-during-mid-training-improves-reasoning-but-slows-factual-recall-ovkjarwnb)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Knowledge distillation during mid-training improves reasoning but slows factual recall","url":"https://daily.dev/posts/knowledge-distillation-during-mid-training-improves-reasoning-but-slows-factual-recall-ovkjarwnb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/knowledge-distillation-during-mid-training-improves-reasoning-but-slows-factual-recall-ovkjarwnb"},"datePublished":"2026-09-02T05:47:03.382Z","dateModified":"2026-10-15T02:22:28.543Z","description":"A study on knowledge distillation finds that the standard forward KL distillation objective behaves differently depending on training phase. During...","image":"https://pbs.twimg.com/media/HRMGmN4awAAO55N.jpg","thumbnailUrl":"https://pbs.twimg.com/media/HRMGmN4awAAO55N.jpg","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/knowledge-distillation-during-mid-training-improves-reasoning-but-slows-factual-recall-ovkjarwnb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,data-science","timeRequired":"PT1M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"Knowledge distillation during mid-training improves reasoning but slows factual recall"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/knowledge-distillation-during-mid-training-improves-reasoning-but-slows-factual-recall-ovkjarwnb#faq","mainEntity":[{"@type":"Question","name":"Does knowledge distillation during mid-training hurt factual recall in language models?","acceptedAnswer":{"@type":"Answer","text":"Yes, forward KL distillation applied during mid-training slows factual recall acquisition even though it continues to improve reasoning ability. This differs from pre-training, where forward KL distillation improves both reasoning and factual recall simultaneously. The effect means a distilled student model tends to absorb the teacher's reasoning patterns more than its factual knowledge during this phase. daily.dev surfaces research like this for engineers tuning distillation pipelines before scaling training runs."}},{"@type":"Question","name":"What is Switch Distillation and how does it fix knowledge distillation problems?","acceptedAnswer":{"@type":"Answer","text":"Switch Distillation is a mid-training objective that applies distillation loss only on tokens where the teacher model shows high predictive confidence, using teacher entropy as the routing signal. On tokens where the teacher is uncertain, it falls back to standard cross-entropy loss instead of forcing the student to imitate the teacher, since uncertain tokens are where factual knowledge transfer tends to break down. Teams weighing distillation strategies can track approaches like this on daily.dev before committing to a training recipe."}}]}
```

