The reality of costs at scale
An AI assistant deployed to 10,000 active users, averaging 20 messages/day at 500 input tokens and 200 output tokens, generates around 70 million tokens daily. At $15/M tokens for a frontier model, that is $1,050/day — $32,000/month. Token cost optimization is not a late-stage luxury; it is an economic viability condition that must be baked into the architecture from day one.
Prompt caching and compression
The first line of savings: native prompt caching. Claude and GPT-4 let you cache identical prefix tokens — cached tokens are billed at 10% of normal price (Claude) or nothing on subsequent calls. The key architectural principle: place all static content (system instructions, few-shot examples, reference documents) at the top of the prompt, variable content at the end. Prompt compression tools like LLMLingua can reduce a prompt by 40–60% while preserving 95% of performance.
Intelligent model routing
Not every call needs a frontier model. A lightweight classifier (fine-tuned GPT-4o-mini or an SVM trained on annotated logs) can route 60–70% of requests to cheaper models — Haiku, GPT-4o-mini — without perceptible quality degradation. On routed requests, cost reduction reaches 85–95%. The classifier cost itself is negligible against the savings realized past 50,000 calls/day.
Semantic cache and batch API
A semantic cache (Redis + embeddings) intercepts similar requests before they reach the API: 15–25% hit rate typical for use cases with recurring questions (FAQ, support, document search). For non-urgent async processing, the Anthropic Batch API and OpenAI Batch offer 50% unit cost reduction. Finally, enforce strict JSON responses (Structured Outputs): this eliminates verbosity, reduces output tokens by 20–30%, and simplifies downstream parsing.