Testing language models' understanding of linguistic features
Published on:
11 July 2023
Primary Category:
Computation and Language
Paper Authors:
Ester Hlavnova,
Sebastian Ruder
Bullets
Key Details
Proposes a morphologically-aware framework to generate cross-lingual tests
Tests model capabilities regarding linguistic features in 12 diverse languages
Models do well on most tests in English but poorly on certain features in other languages
Highlights challenges posed by typological differences in multilingual settings
Explore the topics in this paper
AI generated summary
Testing language models' understanding of linguistic features
This paper proposes a new framework to generate tests that evaluate language models' ability to handle diverse linguistic features across languages. The tests target capabilities like negation, numerals, spatial expressions, and comparatives in 12 languages. Models excel on English but struggle on certain features in other languages, showing gaps in cross-lingual generalization.
Answers from this paper
You might also like
Large language models generate knowledge for AI tasks
Language models gain reasoning skills from noisy examples
Evaluating Large Language Models on Controlled Text Generation
Evaluating language models for optimization
Red teaming language models for safety
Benchmarking language model performance across languages
Comments
No comments yet, be the first to start the conversation...
Sign up to comment on this paper