Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

TL;DR

The paper introduces a cost-aware evaluation framework for security agents, measuring performance not just by success rate but by efficiency under fixed resource constraints. It compares offensive and defensive agents on Cybench and Splunk BOTS v1 tasks, revealing that offensive tasks benefit from increased compute while defensive tasks rely more on strategic tool use and data navigation. The results highlight the need for benchmarks that reflect real-world operational costs.

Why It Matters

Security deployments require cost-effective models that balance performance and resource usage. This work shifts evaluation from idealized success to practical efficiency, guiding developers toward models that are viable in real-world security operations. It also identifies key gaps in defensive agent capabilities that must be addressed for broader adoption.

Key Findings

• Offensive security tasks show improved performance with increased inference compute, allowing scaled open-weight models to match proprietary systems at lower cost.
• Defensive security tasks do not scale linearly with compute; success depends more on precise tool usage, telemetry navigation, and selective data enrichment.
• Current benchmarks fail to capture operational efficiency; cost-aware, SOC-native evaluations are essential for assessing real-world utility of security agents.