What Matters in Quality-Aware VLA Pretraining: A Benchmark and Empirical Study

TL;DR

The paper introduces AXIS-Bench, a benchmark for evaluating quality-aware pretraining in vision-language-action models using 5,000 human demonstrations with varying quality. It systematically compares weighting, filtering, and conditioning strategies across different reward definitions and label granularities, showing that episode-level conditioning improves performance over behavioral cloning and that fine-tuning with constant quality tags boosts success rates.

Why It Matters

This work provides a standardized, reproducible framework for assessing how demonstration quality impacts VLA training, enabling more efficient and effective pretraining. The findings guide practitioners in selecting optimal data curation and training strategies, especially for real-world robotic tasks where high-quality data is scarce.

Key Findings

• Episode-level conditioning outperforms plain behavioral cloning across multiple vision-language-action models.
• Filtering is more effective with chunk-level quality labels, while conditioning works better with episode-level labels, though model-specific reversals occur.
• Fine-tuning with constant quality tags improves success rates after pretraining, regardless of whether inference uses tagged or untagged data.