How to Choose an LLM for Production Without Relying on Leaderboards

1. Define what it means to solve the task#

An assistant that drafts text and an invoice extractor can run on the same model, but they need very different acceptance criteria. Before you start testing providers, write down the product contract: what the feature receives, what it returns, how long it is allowed to take, and which actions require human review. A convincing answer is not good enough if it invents an identifier or changes the amount on an invoice.

For an extractor, separate required fields, output format, supporting evidence, and business rules. For a coding assistant, check that the proposed change compiles, respects permissions, and passes the relevant tests. Valid JSON only proves that the response has the right shape. Your application still has to validate references, ranges, and authorization before it executes any action based on that output.

2. Build an eval set that can prove you wrong#

Start from representative examples from your own application that you are authorized to use. Include incomplete inputs, long documents, language with regional variations, and cases where the correct answer is to abstain. Keep a held-out split that you never use to tune prompts. If you optimize and approve against the same questions, you end up measuring how well you memorized your own exam.

• Store the input, expected output, acceptance criteria, and severity for each case. Leave out secrets and any personal data you do not need.
• Label every failure by root cause: retrieval, reasoning, formatting, tool use, or infrastructure.
• Run the same version of the eval set with the same prompt, budget, and tools against every candidate.

It is fine to start with a small, manually reviewed set and grow it with failures you observe in practice. That initial set is useful for discovering problems, not for backing a statistical accuracy claim. Always report the set size, how cases are distributed, and the absolute number of errors, not just a percentage.

3. Calculate cost per accepted result#

The price per million tokens is only one input to the decision. The real cost of a task also includes retries, retrieval, tool calls, and review. A cheap model that forces you to try three times can end up costing more than a model with a higher list price. Use billed tokens whenever the provider reports them, and record input, output, and cached tokens separately.

type Run = { costUsd: number; accepted: boolean };

function costPerAcceptedTask(runs: Run[]) {
  const accepted = runs.filter(r => r.accepted).length;
  const total = runs.reduce((sum, r) => sum + r.costUsd, 0);
  return accepted === 0 ? null : total / accepted;
}

Compare context windows and pricing conditions before you multiply anything. The model pages in the catalog explain whether the price changes with time of day or with prompt length. A normalized cost is useful for a first shortlist; your final budget should come from replaying the workload you actually expect.

Documentation: DeepSeek: pricing conditions ↗ · xAI: Grok 4.6 context and pricing ↗

4. Measure the end-to-end experience#

Time to first token matters in a chat interface; time to task completion matters when the user is waiting for a file or an action. Measure both where each one applies. Break the total down into queue time, retrieval, the model call, and tool calls so you do not end up blaming the wrong component for a slow response.

Do not publish millisecond differences from three requests as if they were a benchmark. Repeat the runs across different time windows, document the region and concurrency you used, and look at the spread, not only the mean. The goal is to know whether the product meets its latency budget, not to win an artificial race.

5. Design for failure before switching providers#

A timeout can arrive after a tool has already written to another system. Retrying the whole conversation without an idempotency key can duplicate that side effect. Separate generating a proposal from executing an authorized action, and enforce limits on attempts, total time, and cost. The fallback model must also pass the same contract validations as the primary one.

Rerun the hard cases against the fallback candidate. If it does not meet the bar, the safe path may be to ask the user for more information or to route the task to human review. Decide which errors allow a fallback and which ones must stop the flow. Automatically switching providers does not fix invalid input or a missing permission.

6. Ship a reproducible decision#

Keep a manifest with the model identifier, date, prompt, eval set version, configuration, and a summary of the results. Pin model versions when the provider offers them and keep track of their lifecycle and deprecation dates. Roll out gradually, with the ability to return to the previous configuration and with metrics that separate model errors from application errors.

• Shortlist two or three candidates based on capabilities, terms, and budget.
• Run the evaluation and manually review the highest-impact errors.
• Approve quality, latency, and cost against criteria you defined before the experiment started.
• Re-evaluate whenever the model, prompt, tools, or data distribution changes.

Documentation: Mistral: model catalog and lifecycle ↗