Finding the Pivots to Repair Broken Source-Test Evolution Using LLMs
Story 18, from arXiv
A new study explores how LLMs can repair outdated tests when source code evolves, showing that combining test failure symptoms, code changes, and context improves repair success. The research also introduces a reasoning taxonomy that enhances LLM-based test repair efficiency.
Key points
- LLMs can repair obsolete tests when source code evolves, improving test suite reliability.
- Combining test failure symptoms, source code changes, and code context leads to better repair outcomes.
- Self-repair rounds significantly improve the effectiveness of LLM-based test repair.
- A new reasoning action taxonomy derived from LLM behavior helps guide more efficient repairs.
- The study uses a benchmark of 500 real-world Java projects to evaluate LLM test repair strategies.
- Applying the findings to Codex improved its repair success rate on hard cases by 66.7%.
Why it matters
Software engineers rely on test suites to ensure code quality. As code evolves, tests can become outdated, leading to unreliable systems. This research provides actionable insights into how LLMs can be used more effectively to maintain and repair tests, reducing manual effort and improving software reliability.
Counterpoint
The study's findings are based on Java projects and may not generalize to other languages or development environments. Additionally, the cost and complexity of implementing LLM-based repair pipelines could be a barrier for smaller teams.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.