Benchmarking large language models on long, multi-image tasks

Benchmarking large language models on long, multi-image tasks

This paper introduces MileBench, a new benchmark to test multimodal large language models on their ability to process long contexts with multiple images. It includes diagnostic and realistic evaluations across comprehension and generation tasks. Results show closed-source models like GPT-4V and Gemini 1.5 perform well, but most open-source models struggle on long, multi-image contexts. As image count rises, the performance gap between closed and open source models widens, highlighting needs to improve multimodal capabilities.