The real cost of running AI features — and how to control it

AI providers charge by usage — roughly, by the amount of text a model reads and writes — and that per-request fee is the obvious cost. But it is not the whole bill. Add the search infrastructure that finds relevant documents for each answer, the safety checks on every request, the storage for logs, the testing runs, any reserved capacity you have committed to — and the engineering hours spent debugging a system that does not fail the same way twice. Counterintuitively, a ‘cheap’ model that needs three attempts and pages of supporting documents per answer can cost more than a premium model that gets it right the first time.

The foundation is measurement. Tag every AI request with which feature it served, which model ran it, and which prompt version was used. If you cannot answer ‘what does one successful user interaction cost us?’, you cannot manage it.

Discipline about what you send the model comes first. Shorter instructions, tighter selection of supporting documents, and fixed answer formats cut spend without changing models at all. Caching is next: when many users ask essentially the same question — FAQs, policy lookups — serving a stored answer instead of generating a fresh one often halves the traffic for support-style features. Then routing: send routine requests to small, cheap models and reserve the expensive ones for the genuinely hard cases.

Beyond that: process work in bulk overnight where no one is waiting on the answer, cap how many steps an AI agent may take before it must stop, and prefer retrieving the relevant page over pasting the whole manual into every request. Each of these is unglamorous. Together they routinely cut bills by half or more.

Teams sometimes skip building test suites to ‘save money’, then spend far more on regressions, emergency fixes, and the bluntest quality fix there is: upgrading everyone to the biggest model. A good test suite lets you prove that a smaller model, a shorter prompt, or more aggressive caching still meets the quality bar — and that evidence is what lets you actually take the savings. It also prevents the most expensive failure mode: a change that looks fine in a demo and quietly triples usage in production.

Budget for testing explicitly. It is far cheaper than unexplained production spend.

Give someone clear ownership of AI cost per use — jointly across product and engineering, because the levers live in both. Set a budget and an anomaly alert for every feature. Review prompt and model changes with the same seriousness as any change that affects your cost of goods sold, because that is exactly what they are. And when a feature cannot hit its margin target at acceptable quality, that is a product decision to make deliberately — not an operations ticket to shuffle.

The AI features that survive are the ones whose cost curve was designed, measured, and continuously tuned — not hoped away after launch.