EleutherAI presents an interim research update on Mechanistic Anomaly Detection (MAD), a technique for detecting when language models switch between normal and anomalous behavior modes. The team fine-tuned Mistral 7B on 'quirky' datasets where models answer accurately for 'Alice' prompts but use fallible heuristics for 'Bob' prompts. They made the problem harder by diversifying trigger names to prevent simple token-matching shortcuts. Key findings: activation-based detectors outperform attribution patching methods (best offline AUROC 0.94 using quantum entropy with activations), combining activations and attribution patching does not significantly improve results, and uniformly high detection accuracy across all datasets remains elusive. The team also applied detectors to adversarial image detection, finding activation-Mahalanobis works on standard classifiers but not robust ones. Future work includes experimenting with sparse autoencoders as a disentangled feature basis for anomaly scoring.
