ModernBERT-Large-Instruct demonstrates that masked language model (MLM) encoders can perform zero-shot classification and multiple-choice tasks using their pre-training MLM head, without task-specific heads or architectural changes. Using a simple FLAN-style instruction tuning recipe on ModernBERT-Large (395M params), the model achieves 43.06% on MMLU — beating smaller LLMs like SmolLM2-360M and approaching Llama3-1B. A surprising finding: accidentally using 'dummy' MLM examples (where masked tokens were incorrectly labeled as [MASK]) outperformed correct MLM regularization, possibly acting as a form of dropout. Results show strong potential for encoders as multi-task learners, with fine-tuned performance matching or exceeding traditional classification heads.
