CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents
Story 11, from arXiv
CrossSWE is a new benchmark testing coding agents' ability to make coordinated changes across multiple repositories. It shows that even the best models struggle with cross-repository consistency, achieving only 58.48% Pass@1.
Key points
- CrossSWE tests coding agents on real-world cross-repository development tasks.
- It includes 171 instances across 165 repositories and 13 programming languages.
- The benchmark highlights the difficulty of maintaining consistency across interdependent repositories.
- Five LLMs were evaluated, with the strongest model achieving 58.48% Pass@1.
- Cross-repository inconsistencies are the most common failure mode (44.2%).
Why it matters
Software development often involves multiple repositories, and maintaining consistency across them is a critical challenge. This benchmark shows that current coding agents struggle with this, suggesting that new tools and techniques are needed to handle cross-repository coordination.
Counterpoint
The benchmark focuses on a specific scenario and may not represent all real-world coding challenges, limiting its generalizability.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.