Mixture-of-Depths (MoD) is a groundbreaking method introduced by researchers from Google DeepMind, McGill University, and Mila. It empowers transformer models to dynamically allocate computational resources, focusing on the most critical parts of the input sequence. MoD-equipped models can achieve similar performance levels as conventional transformers but with significantly reduced computational loads.
