Huawei AI Introduces ‘Kangaroo’: A Novel Self-Speculative Decoding Framework Tailored for Accelerating the Inference of Large Language Models

Huawei AI introduces 'Kangaroo', a novel self-speculative decoding framework that accelerates the inference of large language models. Kangaroo eliminates the need for external drafter models and improves efficiency by using a fixed shallow sub-network as a draft model. It also employs an early-exiting mechanism to further enhance speed and efficiency.