Seesaw:

a system for bootstrapping image searches

Oscar Moll (orm@csail.mit.edu), Manuel Favela, Sam Madden, Vijay Gadepally

Problem: ad-hoc searches on image datasets, without perfect models.

Searching through your own image databases is a basic building block for many downstream tasks.

A common approach to searching images is semantic embeddings, such as CLIP

Pre-trained embeddings are insufficient because accuracy varies widely

Searches can be time consuming or virtually impossible

Seesaw merges text search with region based feedback

Stage 3:

Region based feedback from user

Region based feedback necessitates region based indexing

User study

• Task: to find 10 examples (multiple queries)
• Users consistently completed task faster using SeeSaw than CLIP alone, sometimes by substantial margins.