Demographic bias in LLMs is concentrated in specific neurons that can be located in minutes using contrastive prompt pairs, without retraining. The OptiPFair library uses PyTorch hooks to capture activations at the down_proj_input of MLP layers, ranks neurons by differential activation score, and zeros the top-k neurons. Zeroing just 5 neurons across 4 layers preserves 99.49% of general capabilities (MMLU, ARC, HellaSwag, WikiText) while shifting bias-related outputs. However, the intervention produces three unpredictable outcomes — bias reduction, amplification, or transformation — because the current metric measures activation magnitude but not direction. The key limitation is that a neuron suppressing bias looks identical to one amplifying it. The proposed fix is a signed bias score using asymmetric corpora where the biased vs. neutral response is known in advance. Experiments cover Llama-3.2-1B, Llama-3.2-3B, and Salamandra-2B in English and Spanish across five demographic categories.
Table of contents
Zeroing just 5 neurons is enough to reshape how an LLM responds to demographic promptsThe Method: How to Locate Biased Neurons Without RetrainingPipeline with OptiPFairResults: What Happens When You Remove the Biased Neurons?Get Pere Martra ’s stories in your inboxResources and Next StepsQuestions this post answers
How do I find which neurons in an LLM are responsible for demographic bias without retraining the model?
Bias-responsible neurons can be found by comparing MLP activations at the down_proj_input across contrastive prompt pairs that differ only in a demographic attribute. The absolute per-neuron activation difference, averaged over sequence positions, produces a bias score ranking. In Llama-3.2-1B, the top-scoring neurons cluster in the final layers (especially layer 15), and the entire analysis runs in minutes on a consumer GPU using the OptiPFair library. Engineers auditing open-source LLMs for bias track techniques like this on daily.dev before committing to a mitigation strategy.
What is the capability cost of zeroing bias neurons in Llama-3.2-1B?
Zeroing 5 neurons in Llama-3.2-1B retains 99.49% of general capabilities on average. Specific benchmark retention: WikiText perplexity 99.60%, MMLU 100.06%, ARC-Challenge 98.36%, HellaSwag EN 99.94%, HellaSwag ES 99.74%. The only systematic degradation is a WikiText perplexity increase of 0.03–0.13 points, which is negligible in absolute terms. Teams deploying Llama variants in production weigh trade-offs like these on daily.dev.
Why does removing high-bias-score neurons sometimes amplify bias instead of reducing it?
The current bias score measures the magnitude of differential activation between demographic prompt pairs but not the direction. A neuron with a high score may be suppressing the stereotype rather than amplifying it. Zeroing it removes that suppression, which increases bias. This is why the same 5 zeroed neurons can reduce bias on one prompt, amplify it on another, and transform it into a different form on a third. The fix requires a signed bias score using asymmetric corpora where the biased response is known in advance. Researchers working on LLM alignment find the latest directional bias work on daily.dev.