Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild Demo

Click on any of the examples below and run the demo.
Note that the model initialization, RL planner, and TTA updates are not fully optimized on GPU for this huggingface demo, and hence may experience some lag during execution. We will improve this in the future.
Project Website

Model Inputs

Live Heatmap Output

Taxonomy

Examples
Satellite Image Ground-level Image (optional) Full Taxonomy Name (optional)

The satellite image CLIP encoder is fine-tuned using Sentinel-2 Level 2A satellite image and taxonomy images (with GPS locations) from iNaturalist. The sound CLIP encoder is fine-tuned with a subset of the same taxonomy images and their corresponding sounds from iNaturalist. Note that while some of the examples above result in poor probability distributions, they will be improved using our test-time adaptation framework during the search process.