Apple ML Research presents Categorical Flow Maps (CFMs) scaled to 1.7B parameters trained on 2.1T tokens, demonstrating that continuous flow matching can be a viable alternative to autoregressive language models. The model generates high-quality text in as few as 4 inference steps while maintaining near-data-level token entropy. The work also introduces a likelihood bound for CFMs in the semi-discrete setting, enabling standard LM benchmark scoring, and provides insights on loss weighting and time scheduling challenges at scale.
Questions this post answers
How many inference steps does a Categorical Flow Map model need to generate text at 1.7B scale?
A 1.7B-parameter Categorical Flow Map model trained on 2.1T tokens can generate diverse, high-quality text in as few as 4 inference steps while maintaining near-data-level token entropy. This is achieved by training a base flow model and then self-distilling it into a CFM, enabling accelerated sampling competitive with autoregressive approaches. Researchers evaluating non-autoregressive LM alternatives track results like these on daily.dev.