Apple researchers introduce LINK, a lexical intervention method that improves cross-lingual knowledge transfer during pretraining by swapping randomly selected words in high-resource (English) training data with their translations using bilingual vocabularies. Unlike prior methods, it requires no parallel corpora, translation systems, or extra training stages. Tested across eight languages and five model sizes, LINK yields notable downstream task improvements in target languages and up to a 2x speedup in training to reach equivalent performance.
Questions this post answers
What is the LINK method for cross-lingual knowledge transfer in multilingual language models?
LINK is a data-level intervention that improves knowledge transfer during model pretraining by randomly swapping words in the high-resource (English) portion of training data with their word-level translations using a bilingual vocabulary. It requires no parallel corpora, translation systems, auxiliary models, or extra training stages, and works for virtually any language pair since bilingual vocabularies are cheap to obtain. Explore more research on efficient multilingual model training approaches like this on daily.dev.
How much training speedup does lexical substitution give for low-resource language models?
Lexical substitution using bilingual vocabularies (the LINK method) achieved up to a 2x speedup in training to reach equivalent downstream task performance in target languages, evaluated across eight languages and five model sizes. This makes it a low-cost alternative to methods requiring large parallel datasets or translation systems for improving multilingual model performance. Follow developments in low-resource NLP training efficiency techniques on daily.dev.