SWE-Time: Benchmarking Temporal Repository Understanding
Story 12, from arXiv
SWE-Time is a new benchmark that evaluates how well AI models understand how code evolves over time in repositories. It tests both retrospective and prospective understanding of code changes, revealing significant gaps in current models.
Key points
- SWE-Time evaluates how models reason about repository evolution over time.
- It tests two directions: retrospective analysis of historical changes and prospective understanding of intended changes.
- The RealWorld-to-Verifiable Construction Framework transforms developer queries into testable tasks.
- 15 coding agents were tested on 1,661 cases from 65 GitHub repositories.
- Claude Opus 4.8 achieved the highest score of 61.9 on SWE-Time Avg.
- Historical code evidence and developer intent are critical for accurate understanding.
Why it matters
As software evolves, understanding its history and future intent is crucial for maintenance, debugging, and refactoring. This benchmark highlights how current AI tools struggle with temporal reasoning, which is essential for real-world code understanding.
Counterpoint
The benchmark focuses on GitHub repositories and may not fully represent all codebases or development workflows, limiting its applicability to other environments.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.