KernelAudit: Contract-Aware Auditing of Validators for LLM-Generated GPU Kernels
Story 13, from arXiv
KernelAudit introduces an independent-audit framework to evaluate the quality of validators used in generating GPU kernels, identifying false-reject and false-accept issues in existing validators.
Key points
- Validators for LLM-generated GPU kernels often have flaws in accepting or rejecting candidates based on task contracts.
- KernelAudit constructs an audit oracle to label candidates as PASS or FAIL based on semantic correctness, implementation integrity, and measurement integrity.
- The audit oracle uses reference-calibrated stress tests, execution trace analysis, and timing checks to evaluate candidates.
- Evaluation of six validators and 716 candidates shows nontrivial false-reject rates and high agreement with human judgments.
- The approach is applied across seven corpora, including constructed controls, to assess validator performance.
Why it matters
Software engineers using LLMs to generate GPU kernels need reliable validators to ensure correctness and performance. KernelAudit provides a method to evaluate and improve validator quality, reducing errors in kernel generation workflows.
Counterpoint
The approach relies on human judgments for validation, which may introduce bias or inconsistency in labeling candidates.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.