LinkedIn's engineering team details how they rebuilt the training infrastructure for AI-Powered Job Search using multi-teacher knowledge distillation. The system coordinates a relevance teacher, engagement teacher, and embedding model to train a compact 0.6B student SLM capable of serving ~22,000 requests/sec/GPU. Key optimizations include: migrating to FSDP2 with Hybrid Sharded Data Parallel (HSDP), switching to high-throughput inference engines with continuous batching and paged attention (2X faster teacher inference, 4X larger batches), and introducing offline distillation with per-shard caching on HDFS to decouple teacher inference from student training. The result is an 8X end-to-end speedup — from ~45 hours to under 5 hours — with no loss in model quality. The entire pipeline is unified under a Ray-based framework where online and offline distillation are per-teacher decisions within a single run, enabling same-day experimentation cycles.
