Measuring Data Diversity with Embeddings: A Taxonomy and Benchmark for Diversity Measures

TL;DR

The paper introduces a taxonomy of 21 embedding-based diversity measures and proposes EmbDivBench, a benchmark evaluating diversity along variety, balance, and disparity dimensions. It finds that Graph Entropy outperforms commonly used measures like Mean Pairwise Distance across all dimensions, offering a more robust assessment of text corpus diversity.

Why It Matters

Accurate diversity measurement is essential for improving training data quality, ensuring test set representativeness, and enhancing model generalization. This work provides a standardized framework and empirical guidance for selecting the most effective diversity metrics in NLP workflows.

Key Findings

• A taxonomy categorizes 21 embedding-based diversity measures into four computational families based on their mathematical foundations.
• EmbDivBench evaluates measures across three distinct diversity aspects—variety, balance, and disparity—using both synthetic and real-world text data.
• Graph Entropy consistently outperforms other measures across all three dimensions, making it the most reliable choice for diversity assessment.