SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving

TL;DR

SWE-Serve is a benchmark designed to evaluate AI agents on real-world production inference serving tasks, featuring 53 tasks derived from actual changes in the SGLang project. It assesses agents across multiple layers of the serving stack, including model integration, runtime execution, and API exposure, with rigorous testing that includes end-to-end serving validation and performance gates.

Why It Matters

The benchmark addresses a critical gap by measuring whether AI agents can deliver production-ready code, not just locally correct solutions. This enables the field to track progress toward building agents capable of handling complex, real-world deployment challenges in inference systems.

Key Findings

• The best-performing model configuration achieved a 75% pass rate on the benchmark, indicating significant room for improvement in agent capabilities.
• End-to-end serving tests rejected about one-third of patches that passed all other tests, revealing a substantial gap between local correctness and production readiness.
• Even with strong performance on individual components, agents often fail in integrated, real-world deployment scenarios, highlighting the need for better system-level reasoning.