Beyond Token Importance: Preserving Spatial Scaffolds for Efficient Vision-Language-Action Inference

TL;DR

The paper identifies that current vision-language-action pruning methods fail to preserve spatial structure, leading to performance drops under token constraints. It introduces GeoScaffold, a training-free method that maintains spatial coverage by partitioning images into regions and selecting key tokens via farthest point sampling. GeoScaffold achieves high task success with only 20% of tokens and significantly faster inference.

Why It Matters

Robotic systems often operate under strict computational limits, making efficient inference critical. By preserving spatial scaffolds, the method enables reliable performance with minimal visual input, improving real-time deployment in resource-constrained environments.

Key Findings

• Uniform spatial sampling (Stride) outperforms semantic and random pruning at moderate pruning ratios but fails under tighter constraints.
• The spatial coverage radius—measuring the largest blind spot after pruning—strongly correlates with task success.
• GeoScaffold maintains 93.2% average success rate at 20% token usage by balancing task relevance and spatial coverage through region-based allocation and farthest point sampling.