Per-token model pricing looks straightforward enough on a spreadsheet. Suppose you run an illustrative B2B document-processing pipeline handling 250,000 extraction and retrieval-augmented generation (RAG) jobs per month, where each call sends a 2,200-token system prompt plus 4,300 tokens of retrieved context and returns a 600-token JSON payload. That works out to 1.625 billion input tokens and 150 million output tokens monthly. At sample commercial API rates of $2.50 per million input tokens and $10.00 per million output tokens, clean single-pass runs cost $5,562.50 a month.
Live traffic rarely behaves like a clean single-pass spreadsheet. What happens when an unusual batch of PDFs starts failing JSON validation, and your orchestration wrapper quietly retries malformed outputs up to three times—appending the validation traceback to the context window on every retry? Input token usage climbs behind the scenes while p95 latency stretches out. When the provider invoice arrives at the end of the month, it only shows aggregate totals per API key. Which workflow broke? Which tenant or prompt edit triggered the surge? Without observability inside the application layer, nobody can tell.
What Standard HTTP Monitoring Leaves Out
Conventional APM dashboards tell you that an outbound request returned HTTP 200 OK along with its total duration. With language models, a slow 200 OK can mask several unrelated problems, and fixing the wrong one wastes engineering time.
Streaming latency is a good example. Measuring end-to-end duration blurs together Time to First Token (TTFT) and Inter-Token Latency (ITL). Rising TTFT usually points to oversized input prompts, broken prefix caching, or upstream queue saturation, whereas sluggish ITL comes from decode speed on a heavy model tier. Step one stage earlier in a RAG pipeline and retrieval telemetry matters just as much. If you do not log vector query latency, retrieved chunk counts, similarity score distributions, and zero-hit rates, it is surprisingly easy to keep stuffing ten low-relevance chunks into every prompt, paying extra input tokens to make the answer worse.
Retries and tenant attribution complete the picture. Record finish_reason (stop, length, tool_calls, or content_filter) alongside schema-validation retries, and sum token counts across the entire parent user action rather than logging each HTTP attempt on its own. Tagging spans with tenant_id, workflow_name, and prompt_version—along with input, cached-input, and output token counts—lets budget alerts fire within minutes of a runaway tenant loop instead of weeks later.
OpenTelemetry Spans Without Leaking Sensitive Prompts
When teams add LLM tracing, it is not unheard of for them to swing too far in the other direction and log everything. Dropping a third-party LLM observability SDK into an application with default settings often exports full raw prompts and completions to an external SaaS vendor. In healthcare, financial, or legal systems, your telemetry pipeline just copied sensitive records outside your compliance boundary.
How do you debug latency and token spikes when you cannot ship raw customer text out the door? Well, most of the time you do not need the text at all. OpenTelemetry's gen_ai.* span attributes let you capture model names, latency, finish reasons, and token counts on every call while keeping raw payload capture off by default:
import re
import time
from opentelemetry import trace
tracer = trace.get_tracer("app.ai.inference")
REDACT_RULES = [
(re.compile(r"\b\d{3}-\d{2}-\d{4}\b"), "[REDACTED_SSN]"),
(re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b"), "[REDACTED_EMAIL]"),
]
def scrub_preview(text: str) -> str:
out = text
for pattern, replacement in REDACT_RULES:
out = pattern.sub(replacement, out)
return out[:1000]
def traced_completion(client, *, model: str, messages: list, tenant_id: str, workflow: str, log_preview: bool = False):
with tracer.start_as_current_span(f"gen_ai.chat {model}") as span:
started = time.perf_counter()
resp = client.chat.completions.create(model=model, messages=messages)
elapsed_ms = round((time.perf_counter() - started) * 1000, 1)
usage = getattr(resp, "usage", None)
choice = resp.choices[0]
span.set_attribute("gen_ai.system", "openai_compatible")
span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.response.finish_reasons", [choice.finish_reason or "unknown"])
span.set_attribute("app.tenant_id", tenant_id)
span.set_attribute("app.workflow", workflow)
span.set_attribute("app.latency_ms", elapsed_ms)
if usage:
span.set_attribute("gen_ai.usage.input_tokens", usage.prompt_tokens)
span.set_attribute("gen_ai.usage.output_tokens", usage.completion_tokens)
if log_preview and choice.message and choice.message.content:
span.set_attribute("app.preview_redacted", scrub_preview(choice.message.content))
return resp
Keep the limits of regex scrubbing in mind here. Pattern matching catches structured strings like Social Security numbers or email addresses as a backstop during staging debug sessions, though regex alone will not catch free-form person names or confidential prose. In regulated production tenants, keep payload preview logging turned off altogether.
Practical Cost Controls That Keep Output Quality Intact
Once span metrics show where tokens actually go, most production savings come from five engineering levers rather than aggressive prompt trimming:
| Optimization Lever | Where It Applies | Implementation Detail | Trade-Off to Validate |
|---|---|---|---|
| Prompt prefix ordering for KV caching | Shared system instructions, tool schemas, and few-shot examples (>1,024 tokens) | Place static instructions and tool definitions first and dynamic user or RAG inputs at the very end so provider prefix caches match | Inserting timestamps, user names, or request IDs at the top of the system prompt invalidates prefix caching on every request |
| Constrained structured outputs | JSON extraction, classification, and tool arguments | Use grammar-constrained JSON schema decoding instead of free-text Markdown prompts wrapped in retry loops | Keep schemas shallow and explicit; deeply nested optional unions still benefit from field-level accuracy checks |
| Async batch endpoints | Nightly document indexing, backfills, and non-interactive summaries | Queue offline jobs through batch inference APIs (commonly discounted ~50% vs. synchronous endpoints with 24h turnaround) | Unsuitable for interactive user waits; needs idempotent job-state tracking |
| Tiered model routing | Intent classification, routing, and straightforward extraction | Evaluate smaller fast models against a golden test set of representative tasks, reserving frontier reasoning models for complex synthesis | Requires an automated evaluation harness so model swaps are judged on measured task accuracy rather than guesswork |
| Conversation history windowing | Multi-turn chat assistants | Re-sending every prior turn causes cumulative input tokens across an N-turn thread to grow quadratically; summarize older turns or cap sliding context | Verify critical constraints from turn 1 remain pinned in structured state |
Weighing Commercial APIs Against Dedicated Private Inference
When traffic is modest or spiky across the week, commercial APIs with enterprise zero-data-retention terms keep you from paying for idle GPUs overnight. That equation flips when sustained throughput stays steady across business hours, or when customer contracts bar external prompt processing altogether. At that point, dedicated vLLM instances on reserved cloud GPUs or your own hardware trade variable token bills for predictable capacity. You can compare the sizing and isolation trade-offs in our private AI deployment guide and private AI security guide.