Token Cost Optimization Playbook

Token cost optimization is the highest-ROI engineering work in production AI. The techniques are well-documented, the dollar savings are measurable per request, and the implementation cost is days, not months. This playbook ranks the eight techniques by impact on a typical production workload, with the math, the worked examples, and the failure modes to avoid.

1. The eight techniques, ranked

Ranked by typical cost reduction on a production AI product with a stable workload pattern:

Rank   Technique                       Typical reduction    Implementation effort
1      Prompt caching                  60-80%               Hours
2      Model routing cascade           70-90%               Days
3      Embedding-first retrieval       40-70%               Days
4      Context compression             20-40%               Hours
5      Batch API                       50%                  Hours
6      Output-length capping           10-30%               Hours
7      Semantic deduplication          5-25%                Days
8      Fine-tuning small models        50-90%               Weeks

The percentage ranges describe the savings on the workload component the technique targets. Combining all eight on a single workload produces the multiplicative effect — typically 75-95% total cost reduction on production AI products that started without any optimization.

The right order to implement them depends on your workload. For chat-style products with stable system prompts, start with caching. For multi-route products with varying difficulty, start with routing. For RAG-style products with retrievals, start with embedding-first. Most workloads benefit from at least 4 of the 8 techniques.

2. Technique 1: Prompt caching

Prompt caching reuses input-token activations across requests with the same prefix^[1]. Anthropic charges 0.10x the base rate for cache reads and 1.25x for the initial write; OpenAI charges 0.50x for cache reads with no separate write premium^[4].

The math. For a workload with a 25,000-token stable system prompt reused 10 times per session, caching produces approximately 80% input-token cost reduction at typical hit rates of 70-90%. The break-even point is 2 reads at Anthropic's rates and 2 reads at OpenAI's rates.

Implementation. Identify the largest stable prefix in each route (system prompt + few-shot examples + retrieved documents typically). Place the cache breakpoint after the stable prefix, before any variable content (user input, dates, session IDs). Deploy and measure hit rate using cache_read_input_tokens in the API response. Tune the breakpoint until hit rate is above 70%.

Failure modes. Prefix below the minimum size (1,024 tokens for Sonnet, 2,048 for Haiku) silently disables caching. Variable content above the breakpoint produces 0% hit rate. Frequent model-version changes invalidate cache entries.

3. Technique 2: Model routing cascade

Model routing sends each request to the cheapest model that handles it well. The implementation: a router classifies the request difficulty (often using a small classifier model or rule-based logic), then routes to either a cheap, workhorse, or frontier model.

The math. A workload that uses Claude Sonnet ($3/M input, $15/M output) for 100% of requests can typically be split 60/30/10 across Haiku/Sonnet/Opus with no measurable quality loss on the easy 60%. Cost reduction:

Workload pattern        Total cost (Sonnet only)    Total cost (cascade)    Reduction
1M input + 200k output   $3,000 + $3,000 = $6,000    $1,260                  79%
- 60% Haiku ($0.80/M)    -                            $480 + $480 = $960
- 30% Sonnet             -                            $900 + $900 = $1,800
- 10% Opus               -                            $1,500 + $1,500 = $3,000

Wait, that totals more. Let me recalculate the cascade:
- 60% Haiku:  600k * $0.80/M + 120k * $4/M  = $480 + $480 = $960
- 30% Sonnet: 300k * $3/M + 60k * $15/M     = $900 + $900 = $1,800
- 10% Opus:   100k * $15/M + 20k * $75/M    = $1,500 + $1,500 = $3,000
Cascade total: $5,760 — only 4% reduction without smarter routing

Better cascade with skewed difficulty:
- 80% Haiku, 18% Sonnet, 2% Opus produces 50-60% reduction

The skew matters. Cost reduction from routing only works when the difficulty distribution is heavily skewed toward the easy end. Most production workloads are exactly that — 70-85% of queries are easy classification, simple extraction, or short generation that runs fine on Haiku or GPT-4o-mini at 5-10x lower cost than the workhorse tier.

Implementation. Build a difficulty classifier (a small model or a rule-based heuristic). Tag each request with its tier. Route to the cheapest model that meets a quality threshold; escalate to higher tiers only on failure. Measure quality on a held-out eval set per tier; adjust the routing rules until quality holds.

4. Technique 3: Embedding-first retrieval

Embedding-first retrieval reduces the input token count by retrieving only the relevant context instead of stuffing everything into the prompt. For RAG and long-context applications, this is the highest-impact technique after caching.

The math. A documentation Q&A product that includes a full 50,000-token document in every prompt costs $150 per 1,000 queries on Sonnet. The same product using embedding-first retrieval to extract the top 5 most relevant 500-token chunks per query costs $7.50 per 1,000 queries, a 95% reduction on input cost. Embedding cost (OpenAI text-embedding-3-small at $0.02/M tokens^[6]) is negligible.

Implementation. Embed the document corpus once; store in pgvector or Pinecone. For each query, embed the query, retrieve the top K (typically 3-10) chunks, include only those chunks in the prompt. Quality typically holds or improves because the model sees less irrelevant content.

Failure modes. Retrieval that surfaces wrong or partial chunks produces worse outputs than including everything. The fix is to invest in retrieval quality — chunking strategy, embedding model choice, retrieval diversity (MMR re-ranking). Embedding-first retrieval works only if the retrieval quality is good; bad retrieval saves money but loses customers.

5. Technique 4: Context compression

Context compression reduces the size of in-prompt content by summarizing, removing redundancy, or applying domain-specific shrinking. Common patterns:

• Conversation summarization. Long chat histories are summarized down to the last 2-3 turns plus a running summary. Saves 60-90% of tokens on long-running conversations.
• Code snippet compression. For coding workflows, comments are stripped, imports are condensed, repeated patterns are referenced. Saves 20-40% in code-heavy contexts.
• Document section selection. Long documents are pre-processed to extract only the relevant sections. Different from retrieval, this is rule-based selection within a single document. Saves 50-80% on structured documents.
• JSON shrinking. Verbose JSON output is replaced with shorter formats (CSV, YAML, abbreviated keys) where the model output is consumed by downstream code. Saves 30-50% on tool-using workflows.

Implementation. Identify the largest variable-content blocks in your prompts. Build a compression step (rule-based or model-based) that runs before the main inference. Measure quality on the same eval set used for routing.

Pattern note. The compression step itself usually runs on a cheap model (Haiku or GPT-4o-mini). Compression cost is typically 5-15% of the savings produced — net win in nearly all cases.

6. Technique 5: Batch API

Batch APIs from Anthropic and OpenAI offer 50% off in exchange for accepting up to 24-hour latency^[2][3]. The discount applies to both input and output tokens.

The math. For workloads that do not need real-time response — overnight document processing, weekly report generation, periodic embeddings, training data generation, the batch API produces a flat 50% reduction. For interactive workloads, batch is unavailable.

Implementation. Identify which workloads are batch-eligible. Typical candidates: scheduled summaries, overnight content generation, weekly analytics processing, training data preparation. Submit batches via the provider's batch endpoint; receive results within the SLA window.

Failure modes. Batch jobs occasionally fail or are returned incomplete. Production batch processing requires retry logic and partial-result handling. The 24-hour SLA is a maximum, not a typical; most batches complete in 1-4 hours.

7. Technique 6: Output-length capping

Output tokens cost 4-5x what input tokens cost (Claude Sonnet at $3/M input vs $15/M output^[5]). Capping output length where possible is a direct cost saving with limited quality impact for many workloads.

Patterns:

• Set max_tokens conservatively per route. A summarization route does not need 4,000 tokens of output; cap at 500. A classification route needs maybe 50 tokens of output; cap there.
• Prompt for brevity. "Respond in 100 words or less" or "Respond in JSON only, no explanation" reliably reduces output by 30-50% on most models.
• Use structured outputs. JSON-schema outputs are typically 30-50% shorter than free-form prose for the same information.
• Strip post-generation. Some routes generate output that is trimmed to a smaller portion before display (e.g., extract one field from a JSON response). Move the extraction into the prompt to avoid generating the discarded fields.

Output capping is the most-skipped technique because the savings per request are small. The cumulative effect at production scale is meaningful — 10-30% on output costs across a workload.

8. Technique 7: Semantic deduplication

Semantic deduplication identifies near-duplicate requests and serves them from a cache rather than re-querying the model. The cache is keyed by semantic similarity (embedding distance) rather than exact string match.

The math. For workloads with significant query repetition (FAQ-style assistants, common-question tools, search products with popular queries), 5-25% of requests are near-duplicates of recent prior requests. Deduplication serves these from cache at near-zero cost.

Implementation. Embed each incoming query. Check the recent-cache for queries within a similarity threshold (typically cosine similarity > 0.92-0.96). If a match exists, return the cached response. Cache TTL is workload-dependent — minutes for fast-changing data, days for stable Q&A content.

Failure modes. Two semantically-similar queries may have different correct answers (e.g., "show me sales for Q1" vs "show me sales for Q2"). Tune the similarity threshold per workload; lower thresholds save more but produce more wrong-answer cache hits.

9. Technique 8: Fine-tuning small models

Fine-tuning a small model on your specific task replaces frontier-model inference with a smaller fine-tuned model that handles the same task at 10-100x lower per-token cost^[8].

The math. A classification task running on GPT-4o ($5/M input, $15/M output) can typically be replaced by a fine-tuned GPT-4o-mini ($0.15/M input, $0.60/M output for the base model, plus fine-tuning costs) at quality comparable to GPT-4o on the specific task. Cost reduction: 30-50x on the classification volume.

Fine-tuning cost. Training cost is typically $50-$500 for a 5,000-10,000 example dataset. Hosted inference cost on the fine-tuned model is roughly 1.5-2x the base small-model rate. Break-even on the training cost happens at relatively low volumes — 100k-500k requests typically.

Implementation. Generate a high-quality training set from current frontier-model outputs (5,000-10,000 examples is typically sufficient). Fine-tune the small model. Evaluate against held-out test set. Route the specific task to the fine-tuned model; keep frontier model only for the long-tail of difficult cases.

Failure modes. Fine-tuned models drift from base capability — they may handle the trained task well but fail on adjacent variations. Fine-tuning is best for narrow, well-defined tasks (classification, extraction, format conversion) and worst for open-ended generation. Re-train periodically as task distribution shifts.

10. The combined effect: a worked example

A solo founder runs an AI documentation assistant on Claude Sonnet 4. Monthly stats:

• 200,000 requests per month
• Average 30,000 input tokens per request (system prompt + few-shot + retrieved docs)
• Average 500 output tokens per request
• Baseline cost: 6B input * $3/M + 100M output * $15/M = $18,000 + $1,500 = $19,500/month

Applying the playbook:

Step                              Cost            Reduction
Baseline                          $19,500/mo      -
+ Prompt caching (75% hit rate)   $7,800/mo       60% reduction on input
+ Model routing (60% Haiku)       $4,200/mo       46% reduction (compounding)
+ Embedding-first retrieval       $2,800/mo       33% reduction (smaller context)
+ Context compression             $2,200/mo       21% reduction
+ Output capping (avg 250 tok)    $1,800/mo       18% reduction
+ Semantic dedup (15% hits)       $1,550/mo       14% reduction
+ Batch API on overnight 30%      $1,300/mo       16% reduction
+ Fine-tuned classifier on easy   $850/mo         35% reduction
TOTAL                              $850/mo         96% total reduction

The cumulative effect of all eight techniques on this workload is a 96% cost reduction — from $19,500/month to roughly $850/month. The implementation cost is approximately 4-6 weeks of one engineer's time spread across the techniques. Payback is roughly 2-4 weeks of operating cost.

Not every workload achieves 96%. Workloads without stable prefixes do not benefit from caching. Workloads without difficulty variance do not benefit from routing. Workloads with strict latency requirements cannot use batch API. The realistic floor for production AI workloads with no optimization is 50-70% cost reduction; the realistic ceiling for fully-optimized workloads is 90-97%.

Two implementation principles. First, measure before each step. The percentage reductions above are typical, not guaranteed; some workloads benefit more or less. Track input cost, output cost, cache hit rate, and per-tier model usage weekly. Second, never sacrifice quality for cost. Run the same eval suite against every change; if quality regresses, the change is not worth the savings. Cost optimization that loses customers is more expensive than the cost it saves. Token cost optimization is engineering, not magic, the techniques are well-documented and the dollar math is verifiable. Solo founders who treat inference cost as a managed engineering surface, not as a fixed bill from the vendor, run AI products at margins that compete with traditional SaaS instead of being squeezed by foundation-model economics.