Patching the Protocol: A Taxonomy of Pre-Registration Repairs in LLM-Proposer Experiments
Story 12, from arXiv
This paper explores how LLM-proposed experiments require protocol repairs, or 'patches,' and introduces a framework for categorizing and auditing these changes.
Key points
- Pre-registration is seen as a trust mechanism for LLM-generated research, but real-world implementations often require protocol repairs.
- Three case studies show that protocol repairs typically involve changing design parameters before data collection, not judgments or data itself.
- A two-axis taxonomy is proposed to classify patches by defect type (environmental, feasibility, correctness) and timing (pre-data, post-data).
- A verifiable non-impact claim is introduced for post-data repairs to ensure transparency.
- A nine-item checklist is distilled from the case studies, all of which were used in practice, not validated as sufficient.
- The credibility of LLM-proposer studies depends on the audibility of patches, not just their occurrence.
Why it matters
As LLMs become more central to research and development, understanding how and when their proposed experiments need adjustments is critical for maintaining trust and reproducibility. This work provides a framework for auditing these changes, which is essential for engineers and researchers using AI tools.
Counterpoint
The study is limited to three case studies, which may not represent the full range of scenarios where LLM-proposed experiments require repairs.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.