A new executable benchmark offers the power engineering community its first standardized method for evaluating AI agents on realistic power systems tasks. Developed to address the gap between academic prototypes and practical deployment, the Power Systems Agent Benchmark enables objective assessment of how well language models and AI agents can solve structured engineering problems.
Unlike traditional text-based benchmarks that grade prose responses, this framework requires agents to return structured solutions that are automatically verified against engineering constraints. An evaluator recomputes relevant quantities—such as power flows and voltage magnitudes—checks operational feasibility, and provides explicit feedback on any constraint violations. This executable evaluation approach mirrors rigorous methods already established in software engineering but is new to power systems.
The benchmark encompasses 41 task families across eight core disciplines: power flow analysis, protection systems, stability assessment, microgrid operations, reliability evaluation, power quality, forecasting, and related areas. Each task derives from cited standards, published formulations, or documented engineering practices, ensuring relevance to industry practitioners.
To prevent contamination from model training data, held-out test cases are synthesized on-demand using private seeds. Task generators produce new instances deterministically, keeping evaluation sets private while remaining inspectable. Initial reference evaluations compared three command-line agents, revealing that the strongest achieved near-ceiling performance while smaller models showed meaningful gaps, with public and held-out scores showing broad consistency.
The research team tested the framework's quality control mechanisms and discovered a latent evaluator bug—validation that the benchmark itself functions as a debugging tool. Notably, the architecture allows upgrading from compact deterministic surrogates to simulator-backed checks without modifying how tasks are posed or solved, enabling gradual sophistication growth.
This benchmark represents a significant step toward standardized AI evaluation in power engineering, potentially accelerating adoption of language models for real-world grid analysis, planning, and operations while maintaining confidence in solution reliability.
