Mislabeling in Bug Localization Benchmarks Threatens Evaluation Validity
Story 15, from arXiv
A study finds that 34.82% of issues in IRBL benchmarks are mislabeled, significantly impacting performance metrics across multiple techniques.
Key points
- IRBL evaluations rely on labeled issues from issue tracking systems, which are often mislabeled.
- Analysis of five benchmarks with over 55,000 issues shows 34.82% mislabeling on average.
- Removing mislabeled issues reduces Mean Reciprocal Rank by 4.68%-21.74% across six IRBL techniques.
- Mislabeling affects performance magnitudes but not relative rankings between techniques.
- The study highlights the need for reliable evaluation data in IRBL research.
Why it matters
As a software engineer using AI for bug localization, you rely on benchmark results to guide your tool choices. If the data used to evaluate these tools is unreliable, your decisions may be based on flawed metrics, leading to suboptimal outcomes.
Counterpoint
The study focuses on issue mislabeling, which may not fully represent real-world scenarios where labeling errors are less frequent or where other factors like code complexity play a larger role.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.