WTF are evals? How to test agents and the software they use

The article explains how evaluation (evals) are essential for measuring the performance of AI agents and the software they interact with, particularly in ensuring tasks are completed correctly. It introduces a tiered approach to evals—deterministic, one-turn, and end-to-end—based on the number of LLM calls required, emphasizing that repeated testing across multiple runs is necessary to assess reliability, especially when LLM outputs vary. The framework aligns with a test pyramid, advocating for lightweight, fast tests to form the bulk of the evaluation suite.