BOBA: Why Coding Agents Fail in Chained Requests
Story 8, from arXiv
A new benchmark shows coding agents struggle with chained requests, completing only 51% of tasks when requests are linked, compared to 95% when handled individually.
Key points
- BOBA-Bench evaluates ten coding agents on 280 chains of 20 consecutive requests from real open-source projects.
- Agents perform significantly worse on chained requests, with completion rates dropping from 95% to 51%.
- The issue is not worse code, but agents missing mistakes that break subsequent requests.
- Agents are poor at detecting errors in code, correctly identifying only 53% of implementations.
- Adding a single verification step and repair round improves chain completion to 59%.
- Coding agents rely heavily on external verification to avoid building on broken assumptions.
Why it matters
As coding agents become more common, their ability to handle complex, chained tasks is critical. This study shows that without proper verification, their reliability drops dramatically, which could impact real-world software development workflows.
Counterpoint
The study assumes agents cannot self-correct, but some agents may have built-in verification mechanisms or access to external tools that could mitigate these issues.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.