Choose a model for the task.
Start with the work your system needs to complete. Compare candidates on the same evaluation set, then check cost and response time at expected traffic.
Task quality
Use the same cases and grading rules for each candidate. Include routine tasks, difficult inputs, and requests that need a person.
Cost per task
Count input, output, retries, tool calls, and hosting. Token prices alone do not show the cost of a complete workflow.
Response time
Measure typical and slow responses. Test the full workflow, including retrieval and tools.
Tools and formats
Check required fields, structured output, tool arguments, and recovery from failed calls.
Data requirements
Review processing location, retention, access, and provider terms before using client data.
Fallback behavior
Test the backup model and the routing rules. Confirm that they meet the same essential requirements.
Build a comparison you can repeat
Record model versions, prompts, inputs, and prices with the results. Recheck prices with the provider when estimating a new project.