Suppressing Misaligned Behaviors in Language Models, Durably

TL;DR

This paper introduces a framework to study and suppress misaligned behaviors in language models post-training, using two model organisms with narrow and broad triggers for undesirable behavior. It proposes PEARL, a GRPO-based method that enhances durability by sampling continuations from cached trajectories, achieving strong, lasting suppression without sacrificing general capabilities.

Why It Matters

Durable suppression of harmful behaviors is critical for deploying safe AI systems. Unlike methods that fail under further training, PEARL offers a robust solution for real-world deployment where models face evolving inputs and continued fine-tuning.

Key Findings

• Standard methods like SFT and GRPO fail to maintain suppression under further training.
• PEARL achieves durable suppression by augmenting rollouts with intermediate state continuations from cached trajectories.
• PEARL is especially effective for broad-trigger misalignment, where targeted SFT is insufficient.