We use a few cookies

Essential cookies keep the library working. With your permission, we'd also like to use analytics to see what's useful and improve the experience. Read our Privacy Policy or Cookie Policy.

Skip to main content
Home/e-Resources/Artificial Intelligence & Machine Learning/Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages
Conference ProceedingsArtificial Intelligence & Machine Learning

Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages

Checking your access to this record.

Abstract & Overview

Pretrained multilingual language models have been shown to work well on many languages for a variety of downstream NLP tasks.However, these models are known to require a lot of training data.This consequently leaves out a huge percentage of the world's languages as they are under-resourced.Furthermore, a major motivation behind these models is that lower-resource languages benefit from joint training with higher-resource languages.In this work, we challenge this assumption and present the first attempt at training a multilingual language model on only low-resource languages.We show that it is possible to train competitive multilingual language models on less than 1 GB of text.Our model, named AfriBERTa, covers 11 African languages, including the first language model for 4 of these languages.Evaluations on named entity recognition and text classification spanning 10 languages show that our model outperforms mBERT and XLM-R in several languages and is very competitive overall.Results suggest that our "small data" approach based on similar languages may sometimes work better than joint training on large datasets with high-resource languages.Code, data and models are released at https://github. com/keleog/afriberta.

Cite this record

Kelechi Ogueji, Yuxin Zhu, Jimmy Lin (2021). Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages. Workshop on Multilingual Representation Learning (MRL). https://doi.org/10.18653/v1/2021.mrl-1.11

Copy in another style

Ogueji, K., Zhu, Y., & Lin, J. (2021). Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages. Workshop on Multilingual Representation Learning (MRL). https://doi.org/10.18653/v1/2021.mrl-1.11
Author names are used as deposited, so check their order before you submit.

Metadata is harvested from open-access indexes and can be incomplete. Check a citation against the published version before relying on it — see our open access & attribution policy.