LayerSkip: An End-to-End AI Solution to Speed-Up Inference of Large Language Models (LLMs)

Researchers explore the possibility of decreasing the layer count for each token in large language models (LLMs) to speed up inference and reduce energy and financial expenditures. They introduce a self-speculative decoding method that combines early departure with speculative decoding, and experiment with layer dropout to minimize computation and increase prediction accuracy.