EngineAgent: Agentizing On-Device LLM Inference Engines
Story 6, from arXiv
EngineAgent is a new system that automates enabling LLMs on mobile devices by managing cross-layer dependencies and hardware-specific workflows, achieving higher success rates than existing coding agents.
Key points
- EngineAgent separates reusable engine and hardware knowledge from model-specific code.
- It constructs a hierarchy-aware engine index to capture cross-layer dependencies.
- Before enabling a new model, it pre-validates workflows using already supported models.
- A builder coordinates cross-layer changes, while an auditor ensures implementation completeness.
- EngineAgent achieves 75% end-to-end success across 12 tasks, versus 33.3% for a coding-agent baseline.
- It reduces per-task time and model-API cost compared to baselines.
Why it matters
For software engineers working with on-device LLMs, EngineAgent offers a more efficient and reliable way to integrate new models, reducing the complexity and cost of cross-layer system changes.
Counterpoint
EngineAgent's success depends on pre-existing supported models and workflows, which may limit its applicability to new or niche hardware and inference engines.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.