Compare AI models on your own tasks. Measure answer quality, cost per task, and response time, then test routing and fallback options.
A new model release raises a practical question: should your team switch? An evaluation set, or eval set, gives you a repeatable way to compare candidates on your own work.
Leaderboards measure general capability on shared benchmarks. A product needs something narrower: a model that does a specific task, to a specific quality bar, at a cost and response time the business can accept. Those are different questions, and a model that wins the first can lose the second. A frontier model that leads on reasoning benchmarks may be slower and many times more expensive than a small model that handles your classification step just as well.
So the unit of decision is not “which model is best.” It is “which model, or combination of models, meets this task’s bar at acceptable cost and latency.” That question has an answer you can measure. Repeat the comparison when relevant models, prices, or requirements change.
An original diagram of a routed model setup with escalation and fallback.
Define the task and the bar first
Name the job precisely
“Customer support” is not a task. “Classify an inbound ticket into one of twelve categories and extract the order number” is a task. So is “draft a reply using the retrieved policy and the customer’s order history.” Each has different requirements, and in a real system they may run on different models.
Break the workflow into steps and note what each one needs: reasoning depth, long context, structured output, tool calling, speed, or reliable pattern matching. Many steps in a production agent are simpler than they look, and simple steps rarely need the largest model.
Write the bar before testing candidates
Before running anything, write down what passing means for each step: a minimum quality score on the eval set, a maximum cost per task, and a p95 response-time limit. At least 95% of measured tasks should finish within that limit. If the bar is set after the results come in, it tends to move toward whichever model someone already liked.
Run every candidate against the same set
Same cases, same graders, same scorecard
A comparison only works when the inputs are held constant. Run each candidate on the same eval set, built from real tasks, graded by the same checks, and reported on the same scorecard: success rate by category, cost per task, latency at p50 and p95, and schema and tool-call failure rates.
Give each candidate a fair shot. Models respond differently to the same prompt, so a short adaptation pass for each model (instructions, examples, output format) can matter as much as the model itself. Keep the adapted prompts in version control next to the results.
Treat impressions as hypotheses
Hands-on impressions are still useful. If someone on the team finds a model noticeably better at following formatting instructions, that is a hypothesis worth testing. It becomes a decision only after the eval set agrees.
Do the arithmetic
Cost is tokens times price times volume
The cost of a model choice is easy to estimate and often skipped:
monthly cost = tasks per month × (input tokens × input price + output tokens × output price)
Take a hypothetical workflow that handles 200,000 tasks a month. Each task sends about 3,000 input tokens (instructions, retrieved context, and conversation) and produces about 500 output tokens. The prices below are illustrative, not quotes from any provider.
- A larger model at $3 per million input tokens and $15 per million output tokens costs $0.009 + $0.0075 = $0.0165 per task. That is about $3,300 a month.
- A small model at $0.25 per million input tokens and $1.25 per million output tokens costs $0.00075 + $0.000625, or roughly $0.0014 per task. That is about $275 a month.
That is a twelvefold difference. If the small model passes the bar on this step, the gap pays for a lot of engineering. If it does not, the savings are imaginary, because failures turn into rework, escalations, and lost trust.
Count what the formula hides, too. Retries add calls. An agent loop adds the input and output tokens from every call; later calls may also repeat earlier context. Retrieved context can grow unless you limit how much each call receives. A number that was right at launch can drift upward quietly.
Latency and quality pull in different directions
Larger models tend to be slower, especially at the tail. For an overnight job that summarizes documents, that rarely matters. For a voice agent, where a long pause reads as hesitation, it can decide the architecture.
Put the latency budget next to the quality bar and the cost ceiling, and you have a triangle. Measure all three together. Routing, prompt size, caching, and tool design can change the tradeoffs.
Use more than one model
Route the easy cases to a small model
For many workloads, the best answer is a combination. A small fast model makes the first attempt. A check decides whether the result is good enough: the output parses, required fields are present, a confidence signal calibrated against labeled cases clears a threshold, or the request does not match a known hard pattern. If the check fails, the task escalates to a larger model.
When most tasks are routine, this keeps average cost and latency close to the small model while holding quality close to the large one. Your eval set tells you whether that is true for your mix. Run the routed setup as its own candidate and compare it to each model alone.
The check has to be cheap and trustworthy. A router that escalates too often saves nothing. A router that keeps hard cases on the small model hurts quality exactly where it counts.
Plan for outages with tested fallbacks
Providers have outages, rate limits, and degraded periods. A production system needs a fallback: a second model, ideally from a different provider, that has already passed the same eval set at an acceptable level. A cascade is only safe if every step in it has been tested. Learning during an incident that the backup model breaks your output schema is an expensive way to find out.
Look past raw quality
Structured output and tool calling are selection criteria
For agents, the most common failures are often not wrong answers. They are malformed JSON, missing fields, invented tool arguments, and tool calls in the wrong order. Measure these separately. A model that writes slightly better prose but fails schema validation more often is usually the worse choice for an agent step, because every failure means a retry or a broken workflow.
Context window versus retrieval
Long context windows make it tempting to put everything in the prompt. Sometimes that is right, as with a single long contract that has to be read as a whole. Often it is not. Larger prompts cost more on every call, slow the response, and can bury the passage that matters. Retrieval that sends only the relevant sections usually wins on cost and latency, and the eval set will show whether it gives up anything on quality.
Data handling can decide for you
Some constraints remove candidates before any test runs. Where is the data processed? How long does the provider retain prompts and outputs? Is zero data retention available on the plan you use? Does the agreement cover the kind of data you plan to send? An AI gateway helps by centralizing provider access, logging, and policy in one place, but it cannot change a provider’s terms. Settle these questions first, then evaluate the models that remain.
Stay able to switch
Put an abstraction layer in front of providers
Model choice can change as the task and available models change. Keep the cost of switching visible. Call models through a gateway or an SDK that normalizes providers, to reduce provider-specific integration work. Tool behavior and prompts can still need changes. Log every call with the model, token counts, cost, and latency, so the next comparison starts from real traffic.
Accept that prompts are partly model-specific
An abstraction layer removes plumbing lock-in, not prompt lock-in. A prompt tuned for one model often needs adjustment for another. Treat per-model prompt variants as normal, keep them versioned, and let the eval set tell you when a variant is good enough.
Re-run the comparison on a schedule
Because models and prices change so often, a model decision has a shelf life. Re-run the comparison on a regular cadence (quarterly works for many teams) and whenever a relevant model ships or a provider changes pricing. An existing evaluation set and test runner make these comparisons easier to repeat.
A practical starting point
Model the decision before you build
Start with the numbers. Estimate task volume, tokens per task, model calls per task, and the latency budget for each step before choosing anything. We built Flowcost for this stage: it models an AI workflow’s cost, latency, and infrastructure tradeoffs before a team commits to an architecture.
Then run the comparison:
- Break the workflow into steps and write the bar for each: quality, cost per task, and p95 latency.
- Build or reuse an eval set from real tasks for each step that matters.
- Pick three or four candidates across sizes and providers, including at least one small model.
- Adapt the prompt for each model, run the set, and fill in the scorecard.
- Test a routed setup and a fallback as candidates in their own right.
- Put the winner behind a gateway, log every call, and put the next comparison on the calendar.
Keep the scorecard with the model configuration and prompts. The next comparison should start from the last measured result.