A.S.E 2.0: How Repository Context Affects Security of LLM-Generated Code
Story 10, from arXiv
A new benchmark, CASE-Bench, evaluates how repository context influences the security of code generated by LLMs, revealing significant gaps in secure code production.
Key points
- CASE-Bench is a benchmark of 145 real-world tasks from CVEs in 51 projects.
- Each task includes Docker environments, functional tests, and exploit validation.
- 13 context construction methods and six LLMs were tested for security outcomes.
- Functionality-focused context improves functional pass rates but not security.
- Withholding safeguards reduces secure pass rates by up to 8.1 percentage points.
- Coding agents often fail to observe critical security safeguards in their trajectories.
Why it matters
As LLMs generate more code, understanding how repository context influences security is critical for building safer systems. This study highlights how context construction can either protect or undermine code security.
Counterpoint
The study focuses on repository-level tasks and may not fully represent all code generation scenarios, especially those not tied to real-world CVEs.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.