Task-Induced Riemannian Metrics for Vision Transformer Feature Spaces

TL;DR

This paper introduces task-induced Riemannian metrics for Vision Transformer feature spaces, providing a more accurate representation of the task geometry. The authors propose a diagnostic tool and a Spectral Pullback Network to learn low-rank approximations of the metric. This approach improves token importance prediction and geometric token pruning.

Why It Matters

The proposed method enhances the understanding of Vision Transformer feature spaces and their geometric properties, leading to improved performance in various computer vision tasks. This research has practical significance in applications such as image classification, object detection, and depth estimation. The developed techniques can be applied to various Vision Transformer architectures, making them more efficient and effective.

Key Findings

• Task-induced Riemannian metrics can be learned using a low-rank approximation
• A matrix-free diagnostic tool can predict the viability of learned-metric architectures
• The Spectral Pullback Network improves token importance prediction and geometric token pruning