CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels

TL;DR

The CORDIAL method calibrates ordinal outputs from large language models using a small channel of interpretable parameters. This approach corrects noisy model outputs with a handful of labels. It achieves state-of-the-art results on several datasets and models.

Why It Matters

Accurate calibration of language models is crucial for real-world applications, such as sentiment analysis and text classification. CORDIAL's ability to calibrate models with few labels makes it a practical solution for low-resource scenarios. This method can improve the reliability of language models in various industries.

Key Findings

• CORDIAL outperforms other calibrators with 5-100 labels
• it matches strong baselines with significantly fewer labels
• it enables learning priors from other tasks and fusing multiple models