ExpertCrowd-SWE: Benchmarking Coding Agents on Expert-Authored Tasks
Story 9, from arXiv
ExpertCrowd-SWE is a new benchmark with 100 software engineering tasks for evaluating coding agents. It includes real-world codebases, reference solutions, and graders that mimic maintainer feedback.
Key points
- The benchmark includes 100 tasks across 87 open-source projects and 11 languages.
- Each task is a change request paired with a reference solution and a grader that awards partial credit.
- Scores are based on weighted, executable criteria and include pass/fail gates.
- Seven frontier models were evaluated, with Claude Opus 5.5 and GPT-6 Astra leading.
- No model solved all tasks, and scores vary widely across models and runs.
- The benchmark includes adversarial patches and alternative control solutions.
Why it matters
This benchmark provides a realistic measure of how well AI coding agents can handle real-world, complex software engineering tasks. It highlights the current limitations and performance gaps between top models.
Counterpoint
The benchmark focuses on tasks from open-source projects, which may not fully represent the diversity of codebases or the complexity of proprietary systems.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.