RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

TL;DR

RRSI introduces regularization into recursive self-improvement of LLM agent harnesses by limiting edit complexity over time and filtering proposals based on utility and cost. It improves performance on both in-distribution and out-of-distribution tasks while reducing computational overhead.

Why It Matters

This approach enables more robust and generalizable agent systems by preventing overfitting during automated self-improvement, making agents more adaptable to new tasks and environments without increased resource usage.

Key Findings

• RRSI reduces overfitting in agent harness evolution by applying temporal annealing to edit budgets and favoring unexplored evolutionary paths.
• The critic-pruner mechanism in the selector eliminates benchmark-specific or redundant changes, promoting reusable and efficient components.
• RRSI achieves up to 14.1 absolute gain on training benchmarks and up to 4.7 gain on out-of-distribution tasks, with 30% fewer policy tokens used.