Paper Image

Evaluating multilingual language models

Published on:

9 October 2023

Primary Category:

Computation and Language

Paper Authors:

Sina Bagheri Nezhad,

Ameeta Agrawal


Key Details

Compares mBERT, XLM-R, GPT-3 on next token prediction across 43 languages

Resource availability strongly correlates with model performance

Complex interplay between language families, word order, script and performance

Script type affects GPT-3 more than mBERT, XLM-R

Word order impacts GPT-3 accuracy but not mBERT, XLM-R

AI generated summary

Evaluating multilingual language models

This paper presents a comprehensive comparative evaluation of three prominent multilingual language models - mBERT, XLM-R, and GPT-3 - assessing their capabilities across 43 languages using a self-supervised next token prediction task. The key findings demonstrate the importance of resource availability and reveal complex relationships between performance, language families, word order, and script type. The study provides valuable insights to guide future research towards enhancing multilingual language model effectiveness across diverse languages.

Answers from this paper


No comments yet, be the first to start the conversation...

Sign up to comment on this paper

Sign Up