A step-by-step guide to fine-tuning ModernBERT, a modernized BERT encoder model, for text classification tasks — specifically building an LLM router that classifies user prompts by difficulty. ModernBERT offers 8192-token context, RoPE embeddings, and alternating attention patterns, achieving ~3x faster training than original BERT (321s vs 1048s) while matching or exceeding its F1 score. The tutorial covers environment setup, dataset preparation using the Hugging Face ecosystem, training with the Hugging Face Trainer including torch_compile and flash attention optimizations, and running inference via the transformers pipeline.
