Direct answer
Before switching models to save money, check the configuration. The expensive mistakes are usually mechanical: a prompt cache that never hits because dynamic content sits in front of the stable prefix, token limits far above what any answer needs, retries that resend whole documents, and a model quietly changed by a leftover setting. Each one is cheap to find in the usage data and cheap to fix.
Prompt caching only works if the prefix is stable
Both OpenAI and Anthropic discount input tokens that match a cached prefix. OpenAI bills cache reads at a fraction of the normal input price, and Anthropic bills them at a tenth of base input on most models. Both require the matched part of the prompt to be identical, from the start up to the cache point.
That last rule is where teams lose the discount. If a timestamp, a user name or a retrieved document appears before the system prompt and tool definitions end, every request looks new and nothing is reused. OpenAI’s guidance is explicit: put stable instructions and shared reference material first, and dynamic or user-specific content at the end.
We saw this in a regulatory research assistant built on a multi-stage pipeline. Its cache hit rate was 0%. The automatic cache point sat after the per-request content, and a second layer stripped cache markers written for another provider. Moving an explicit cache point to the end of the static prefix fixed it: a repeat call then read 2,199 of 2,220 input tokens from cache, and roughly 1,500 to 2,000 input tokens per stage call are now billed at the cached rate. One stage also had to grow past the provider’s minimum cacheable length of 1,024 tokens before it could be cached at all.
Token limits are budgets, not safety margins
A generous max_tokens looks harmless because you pay only for the tokens actually generated. The risk is in what it allows: an output that is permitted to run long sometimes will, through a loop, a verbose fallback or a prompt that invites it, and nothing stops it before the bill does.
Set limits from real outputs. In one content pipeline we capped output at 7,000 tokens, sources at 10, each snippet at 700 characters and prior notes at 6,000 characters, and allowed one correction attempt.
Retries should not resend everything
The same pipeline’s correction step sent the rejected draft back as input on every retry, adding up to 50,000 characters to the next request. The fix was to describe what to correct without resending the whole payload, and to quarantine topics that failed three times instead of retrying them forever. A missing-internal-links check had also paid for a second full generation; it now repairs links locally from a verified list of pages.
Pin the model and log raw usage
A leftover configuration value could silently change which model generated content and images, which changes both quality and cost without anyone deciding to. Pin models in code and ignore stray settings. Log raw token counts per call, including cached and cache-write tokens, rather than dollar estimates calculated from hard-coded prices that go stale.
OWASP lists this under LLM10:2025 Unbounded Consumption: cap input sizes, rate-limit per user, time out expensive operations, and monitor usage for unusual patterns.
A cost review in one afternoon
- Pull a day of usage and check cached tokens per call. If they are near zero on repeated prompts, look at what sits in front of the stable prefix.
- Compare each call’s token limit with its real output sizes.
- Look for retries and corrections that resend large inputs.
- Confirm the model each feature uses is the one you chose.
- Cap input size and add per-user rate limits.
- Log raw usage per feature so the next change can be measured.
Switch models after this, if the numbers still say so.
Related Zenveus services: Agentic AI Development and AI Prototype Hardening