Self-supervised image retrieval with open instructions

Self-supervised image retrieval with open instructions

This paper introduces MagicLens, a series of self-supervised image retrieval models that can follow open-ended text instructions to find relevant images. The key insight is that image pairs naturally co-occurring on web pages contain diverse implicit relations beyond visual similarity. By using large language models to make those relations explicit as text instructions, rich training data is created. Experiments show MagicLens matches or exceeds prior state-of-the-art on multiple benchmarks, with 50x fewer parameters on some. Additional analysis finds it succeeds on complex search intents missed by prior methods.