Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild Demo

Click on any of the examples below and run the TTA demo. Check out the multimodal heatmap generation feature by switching to the next tab above.
Note that the model initialization, RL planner, and TTA updates are not fully optimized on GPU for this huggingface demo, and hence may experience some lag during execution. If you encounter an 'Error' status, refresh the browser and rerun the demo. We will improve this in the future.
Project Website

The satellite image CLIP encoder is fine-tuned using Sentinel-2 Level 2A satellite image and taxonomy images (with GPS locations) from iNaturalist. The sound CLIP encoder is fine-tuned with a subset of the same taxonomy images and their corresponding sounds from iNaturalist. Note that while some of the examples above result in poor probability distributions, they will be improved using our test-time adaptation framework during the search process.

Search-TTA: A Multimodal Test-Time Adaptation Framework for Visual Search in the Wild Demo

Click on any of the examples below and run the multimodal inference demo. Check out the test-time adaptation feature by switching to the previous tab above.
Project Website

The satellite image CLIP encoder is fine-tuned using Sentinel-2 Level 2A satellite image and taxonomy images (with GPS locations) from iNaturalist. The sound CLIP encoder is fine-tuned with a subset of the same taxonomy images and their corresponding sounds from iNaturalist. Note that while some of the examples above result in poor probability distributions, they will be improved using our test-time adaptation framework during the search process.