flâneur

AgentIR: Workflow Aware Predictive Scheduling for Agent Graph LLM Serving - Krish Modi | Krish Modi

krishmodi.com · 1,633 words · saved by 1 readers

The fastest LLM request scheduler for agent systems

Loading... 1. Abstract AgentIR is a predictive LLM request scheduler for agent systems. It sits between agent builders and LLM providers, takes user supplied agent system code, compiles it into an intermediate representation, and uses that structure as runtime hints for online scheduling. The goal is simple: treat an agent as a workflow, not as a bag of independent calls, and use that workflow context to reduce end to end latency and improve throughput on the same underlying model servers. The result is scheduling that reduces average graph latency by 41.3% compared to round robin scheduling,

saved by

related reading