Pillar guide
What is an LLM router?
A plain-English definition of the LLM router: how the routing decision works, how a router differs from an LLM gateway or a marketplace like OpenRouter, where the RouteLLM paper and open-source routers fit, and how to stand up a model router for your own task from production traces.
TL;DR
What is an LLM router?
An LLM router is the component that decides which language model handles each request, sending every task to the model that serves it best on quality, latency, and cost. Implementations range from hand-written rules to learned classifiers like RouteLLM. In a production agent, routing is part of the harness — a decision your system owns and encodes — not a property of whichever marketplace or gateway carries the API call.
Ask what an LLM router is and you will get three different artifacts in response: a research system like RouteLLM that learns to choose between a stronger and a weaker model, a feature inside an LLM gateway that load-balances or falls back across providers, and a marketplace like OpenRouter that fronts hundreds of models behind one API. All three are real, and none of them answers the question production teams are actually asking, which is narrower: which model should serve which of our tasks, and what evidence decides that.
This guide covers the definition and the mechanics, the difference between a router and a gateway, what OpenRouter is and is not, where the open-source RouteLLM work fits, and the position this page argues: in a production agent, routing is a harness job — a decision your system owns, made from production traces, and verified before it ships. (If you were looking for a Wi-Fi router, this page is about AI systems, not network hardware.)
Updated
Mechanics
How LLM routers work
Every LLM router — whether it is a hand-written rule or a trained classifier — answers the same question per request: which model is good enough for this task at the lowest acceptable cost and latency? The implementations differ in how that decision gets made.
- Static rules: route by task type, input length, language, or user tier. The simplest AI model router is an if-statement, and for many products it is also the right one — cheap to build, trivial to audit, easy to revert.
- Learned routers: a small classifier trained to predict, per request, whether a cheaper model will produce an acceptable answer, escalating to the expensive model otherwise. RouteLLM is the best-known open example.
- Cascades: send the request to the small model first, then escalate when a verifier, confidence score, or self-check says the answer is not good enough. Trades a second call on hard requests for savings on easy ones.
- Task-level routing: partition traffic by task rather than scoring each request — classification goes to one model, drafting to another, extraction to a third. Coarser than per-request routing, and far easier to evaluate and debug.
Whatever the mechanism — and whatever you call it, an LLM model router and an AI model router are the same thing — the hard part is not the switch statement. It is the evidence behind it: a map of which tasks your traffic actually contains, and a per-task measurement of how candidate models perform on those tasks. A router without that evidence is a cost dial someone set by intuition.
Disambiguation
LLM router vs LLM gateway
The two terms travel together and name different jobs. An LLM gateway is infrastructure: one API in front of many providers, handling keys, rate limits, retries, spend tracking, and fallbacks. LiteLLM and Portkey are the well-known examples. An LLM router is a decision: which model should this request go to in the first place.
The confusion is understandable because gateways ship routing features — load balancing across deployments, fallback chains when a provider errors or times out. But gateway routing is availability routing: it routes around failures. It does not know that your summarization traffic does fine on a small model while your policy-reasoning traffic regresses badly on anything below the flagship. That knowledge is quality routing, it is task-specific, and it has to come from evidence about your own traffic.
In practice the two compose: the gateway carries the call, and the router — a rule or classifier you own, informed by per-task evidence — decides which model the gateway should carry it to. If you are choosing infrastructure, you are shopping for a gateway. If you are deciding which model serves which task, you are building a router, and the gateway cannot make that decision for you.
SourcesBerriAI, LiteLLM (open-source LLM gateway) · Portkey (AI gateway)
Search intent
Looking for an OpenRouter alternative?
OpenRouter is a model marketplace: one API and one bill across hundreds of models from many providers, with its own routing conveniences layered on top. If what you want is another marketplace or unified API, the honest answer is that the alternatives are gateways — LiteLLM if you want open source and self-hosted, Portkey and its peers if you want managed — or simply holding direct provider keys once your model list stabilizes.
Moda is not an OpenRouter alternative in that sense, and this is not a comparison page. Moda is not a marketplace, does not resell inference, and does not sit in your request path.
But a meaningful share of people typing openrouter alternative are not shopping for a bigger menu of models. They already know roughly which providers they trust; what they lack is the evidence for which model should serve which of their own tasks — and the confidence that a cheaper choice will not quietly regress the product. That job is not solved by switching marketplaces. It is a harness engineering job: partition your production traffic by task, measure candidate models on it, encode the routing decision in your own system, and verify it before it ships. That is the job Moda serves.
SourceOpenRouter
Research
RouteLLM and open-source LLM routers
If you searched route llm or RouteLLM, you most likely mean the LMSYS work: RouteLLM is a research paper and open-source framework for learning to route between a stronger and a weaker model. Its routers are trained on human preference data and decide per query whether the cheap model's answer will be acceptable, spending the expensive model only where it matters.
To be explicit: Moda is not RouteLLM and is not affiliated with it. If you want an open-source LLM router to experiment with learned cost-quality routing, RouteLLM is the canonical starting point — the paper, the trained routers, and the framework are all public. Gateways like LiteLLM also ship open-source routing primitives (fallbacks, load balancing) that cover the availability side.
What the open-source routers do not give you is the task evidence. RouteLLM optimizes a general cost-quality frontier over broad preference data; your product's traffic is a specific distribution of specific tasks, and the router that is right for it depends on how models actually perform on those tasks. Whichever routing mechanism you adopt — learned, cascade, or an if-statement — the per-task measurement on your own traces is the input it cannot work without.
SourcesOng et al., RouteLLM: Learning to Route LLMs with Preference Data (arXiv:2406.18665) · lm-sys/RouteLLM on GitHub · LMSYS, RouteLLM announcement
The claim
Routing is a harness job
In a production agent, which model serves which task is a harness decision — the same class of decision as which tools are exposed and what the system prompt covers. The model slot is one component of the harness, and the routing rule that fills it belongs in code you own, made from evidence you can inspect.
The reason is that the decision does not hold still. Models improve, prices move, and your traffic drifts toward tasks nobody designed for. A marketplace default or a gateway fallback chain encodes nobody's judgment about your tasks; a routing rule in your harness encodes yours, is reviewable in version control, and reverts with a rollback when the evidence changes.
The loop that keeps it honest is the standard harness engineering loop applied to the model layer. Cluster production traffic into a task taxonomy, so "which model for which task" has a denominator. Measure how candidate models perform per task on replayed production traces, not on public benchmarks. Encode the routing decision in the harness. Verify the change against replays before it ships, and watch the affected tasks after. Moda is a harness engineering platform, and this is where it sits in a routing project: the task taxonomy from your real traffic, the per-task model comparison on your real traces, and the replay verification before the router changes — while the router itself stays yours, in your harness, out of anyone else's request path.
Practice
Standing up the best LLM router for your task
The best LLM router is not a product you pick off a leaderboard — it is the router calibrated to your task distribution. The buyer job we see is consistent: a team with a few high-stakes tasks wants to stop paying flagship prices for requests a cheaper model serves equally well, without regressing the tasks that matter.
- Partition traffic by task, from production. The routing unit is the task, and the task list has to come from what users actually do — not from the product spec.
- Measure per task, on your traces. Replay real production conversations through candidate models and compare outcomes per task. Public benchmarks rank models on someone else's distribution.
- Start task-level, not per-request. Routing whole task clusters to a model is easier to evaluate, debug, and explain than scoring every request; add per-request routing only where a task genuinely spans difficulty levels.
- Encode the decision in the harness. The routing rule is a versioned artifact with an author and a diff, next to the prompts and tool schemas it interacts with.
- Verify before shipping, watch after. Replay the routing change against production traffic before it ships, then watch the affected tasks for regressions the aggregate metrics will hide.
This is the shape teams like Endera and Stan bring to us: not shopping for a model marketplace, but standing up a model router for their own task — with production traces as the evidence for which model serves it, and replay verification as the reason to trust the switch.
Frequently asked
Questions
What are LLM routers?
LLM routers are components that decide which language model handles each request or task, so that easy or cheap work goes to smaller, faster models and hard or high-stakes work goes to stronger ones. A router can be a hand-written rule, a cascade that escalates on low confidence, or a learned classifier like RouteLLM. The router's job is a quality-cost-latency decision; it is distinct from the gateway or SDK that physically carries the API call.
What is the difference between an LLM router and an LLM gateway?
A gateway is infrastructure: one API in front of many providers, handling keys, rate limits, retries, spend tracking, and fallbacks — LiteLLM and Portkey are common examples. A router is a decision: which model should this request go to in the first place. Gateways ship availability routing (fallbacks, load balancing) that routes around failures; deciding which model serves which task well is quality routing, and it requires evidence about your own traffic that no gateway has.
Is Moda an OpenRouter alternative?
Not in the marketplace sense. OpenRouter is a unified API and marketplace across hundreds of models; Moda is a harness engineering platform and does not resell inference or sit in your request path. If your search is really about deciding which model should serve which of your own tasks — and being confident a cheaper model will not regress the product — that is the job Moda does: task clustering from production traffic, per-task model comparison on replayed traces, and verification before the routing change ships.
What is RouteLLM?
RouteLLM is a research paper and open-source framework from LMSYS (Ong et al., arXiv:2406.18665) for learning to route between a stronger and a weaker model. Routers trained on human preference data predict per query whether the cheaper model's answer will be acceptable, so the expensive model is only used where it adds value. The code and trained routers are public at github.com/lm-sys/RouteLLM. Moda is not RouteLLM and is not affiliated with it.
Is there an open-source LLM router?
Yes. RouteLLM (from LMSYS) is the best-known open-source learned router, shipping trained routers and a framework for cost-quality routing between model pairs. LiteLLM is an open-source gateway whose routing covers the availability side: fallbacks, load balancing, and retries across providers. Neither knows how models perform on your product's specific tasks — that per-task evidence has to come from your own production traces, whichever routing mechanism you run.
What is the best LLM router?
The one calibrated to your task distribution. Rankings of routers, like rankings of models, are measured on someone else's traffic. The practical evaluation is: partition your production traffic by task, replay real traces through candidate models, and see where a cheaper model holds up and where it regresses. For many products the best router is a task-level rule an engineer can read; learned per-request routing earns its complexity only where a single task genuinely spans difficulty levels.
How does Moda help me stand up a model router?
Moda supplies the evidence and the verification; the router stays yours. It clusters production conversations into a task taxonomy so you know what your traffic actually contains, compares how candidate models perform per task by replaying your real traces, and verifies the proposed routing change against replayed traffic before it ships. The routing rule itself lives in your harness as a versioned artifact — Moda never sits in your request path.
For AI agentsThis page as Markdown · llms.txt · Agent skills index · Claude Code skill
Stand up a model router for your task.
Moda is a harness engineering platform: it clusters your production traffic by task, compares how models perform on each task using your real traces, and verifies the routing change against replayed traffic before it ships.