Do Coding Agents Follow AGENTS.md? The Effect of Policy Wording on Instruction Compliance
Story 17, from arXiv
A study shows coding agents do not consistently follow policies in Agent Context Files (ACFs). The effectiveness of policy wording in influencing compliance varied by style and model.
Key points
- Researchers tested how different policy styles (conditional, prescriptive, explanatory) affected coding agent compliance.
- Conditional phrasing led to lower compliance compared to prescriptive and explanatory styles.
- The best-performing policy style varied depending on the model and task combination.
- The study used 86 benchmark tasks across four harness-model combinations.
- The findings highlight the importance of policy wording in guiding AI behavior in software engineering tasks.
Why it matters
Software engineers using AI tools for code generation or automation should be aware that how policies are worded can significantly impact how well agents follow them. This has implications for reliability and predictability in AI-assisted workflows.
Counterpoint
The study focuses on a limited set of policy styles and may not capture the full range of possible influences on agent behavior.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.