DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions

TL;DR

This paper introduces DyMT-ESB, a dynamic multi-turn evaluation of social bias in user-LLM interactions. It reveals social bias in coherent, response-conditioned interactions, including late-emerging and non-monotonic bias patterns. The study highlights the importance of analyzing social bias dynamically.

Why It Matters

Evaluating social bias in LLMs is crucial for ensuring safety and fairness in interactive applications. The findings of this study can inform the development of more nuanced and effective bias mitigation strategies. This research has significant implications for the responsible deployment of LLMs in real-world settings.

Key Findings

• late-emerging bias
• non-monotonic bias patterns
• bias re-emergence