Subscribe

Coding with AI: which model, context size, CI agents and cost

Measure cost per coding task and review what the agent writes

Data reviewed on Sep 8, 2026 · LLM Stats indexes and official prices

The data, today

The context

Picking a model for coding is no longer about autocompleting lines. Agents read the repository, propose changes across several files, run tests and open pull requests. That makes things matter that used to be irrelevant: how much context the model needs to understand your codebase, how many tokens a complete task burns through across all its iterations, and what happens when an agent runs unattended inside your CI pipeline.

As a starting point we use the LLM Stats coding index and each provider's official prices. But the number you care about is a different one: cost per task that is finished and accepted in your repository, retries included. And no metric replaces review. Generated code goes through tests, static analysis and a human before it is merged, exactly like code from any contributor who joined the team last week.

What to decide

  1. 1

    Which model should I code with?

    Filter by the coding index and pick three candidates at different price points. Give them the same ten real tasks from your backlog, each with clear acceptance criteria, and measure how many they finish unaided, how many iterations they need and how many tokens they consume in total. Decide on cost per accepted task.

  2. 2

    How much context does a large repository need?

    Less than you would think: good agents search and read only the relevant files. Measure actual input tokens per task in your trials. If the agent depends on loading whole modules, a bigger window helps, but first check whether a repository map or internal docs achieve the same.

  3. 3

    Should agents run in CI?

    Yes, for bounded tasks: fixing lint, bumping dependencies, writing missing tests or reviewing PRs. Give the agent a least-privilege token with no access to production secrets, and cap time and tokens per run. Track accepted versus discarded PRs and cost per run.

  4. 4

    How do I calculate cost per task?

    Log input and output tokens for every agent step, not just the first call. A task may take dozens of calls with context growing at each one. Add them up, apply official prices and divide by accepted tasks; rejected attempts cost money too.

  5. 5

    How do I review generated code for security?

    Treat it like code from an outside contributor: static analysis, dependency and secret scanning, required tests and a human reading the diff. Pay close attention to new dependencies the model suggests, since they may not exist or may be malicious. Keep track of which findings recur most often.

Common mistakes

  • Choosing a model on price per token without measuring how many calls each task takes.
  • Giving a CI agent credentials with write access to production.
  • Merging generated code because the tests pass, without anyone reading the diff.

Tools and comparators

AI cost calculatorAI models comparator

Guides to go deeper

Want a recommendation for your case?

Tell us your volume and the options you are weighing. We reply in writing with the numbers of your real usage; no commitment.

Request advice

No provider pays for its position. Indexes come from LLM Stats; prices from each provider's standard API. How we measure

Reduce AI costsMeasure tokens per request and pay for big models only when neededAI customer supportWork out cost per conversation without dropping below your quality barDocument search and RAGFind the right passage before asking the model for an answerCut cloud costsFind the hidden spend in egress, operations and idle resourcesHandle traffic spikesAbsorb traffic bursts without dropping requests or the databaseShip to productionDeploy from Git with separate environments and clean configuration
The Codifly brief

A clearer perspective.
In your inbox.

Analysis, guides and new technology comparisons.

You can unsubscribe whenever you like.