Paper Image

Using small language models for evaluating summaries

Paper Authors:

Neema Kotonya,

Saran Krishnasamy,

Joel Tetreault,

Alejandro Jaimes

Bullets

Key Details

Tested small language models for evaluating summaries

Simple prompting strategies worked well

Chain-of-thought prompting aided reasoning

Best model achieved competitive scores

AI generated summary

Using small language models for evaluating summaries

This paper explores using small language models to evaluate the quality of text summaries. The authors experiment with different prompting strategies to get the models to score summaries on coherence, consistency, fluency and relevance. They find that a simple prompt with a rating scale performs well, and chain-of-thought prompting also helps the model reason through its evaluation.

Answers from this paper

Comments

No comments yet, be the first to start the conversation...

Sign up to comment on this paper

Sign Up