Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

TL;DR

The study evaluates how different prompting strategies affect LLM-as-a-Judge performance across 27 languages and 4 tasks, revealing that prompt effects are highly dependent on the judge model, task, and language. Results show that the choice of judge model outweighs prompting strategy, with strategy impacts reversing across judges and deteriorating in low-resource languages.

Why It Matters

This challenges the assumption that prompting strategies are universally effective, emphasizing the need to validate configurations for specific evaluation settings. It highlights risks in cross-study comparisons and calls for transparency in both prompt and judge choices in multilingual AI evaluation.

Key Findings

• The effectiveness of prompting strategies varies significantly depending on the judge model used, with judge choice having a stronger influence than the prompt type.
• Same prompting strategies can yield opposite evaluation outcomes across different judge models, indicating direction reversals in performance rankings.
• Prompting strategy stability decreases in low-resource languages, where aggregated results obscure inconsistent behavior across individual languages.