Total AI cost is more than token price: it is tokens, plus request volume, plus retries, plus failed requests, plus latency overhead, plus fees. Control it in three phases — prevent spend before inference, route to the least expensive capable model, attribute every request afterward.

The nRouter LLM cost optimization framework, in full: https://nrouter.ai/llm-cost-optimization

Disclosure: I work with the nRouter team.

nRouter Docs — Introduction: Getting Started, API Keys and Guides. Documentation is where a gateway proves itself first. The Introduction page explains nRouter in one read: one API key and one bill instead of per-provider credentials and dashboards, guardrails, budgets and rate limits enforced before a request reaches a provider, a five-step request path from authentication to a response with observability headers, the live catalog as the authority on models, and a three-step start that ends with a first OpenAI-compatible API call. Start here: https://nrouter.ai/docs/gettin....g-started/introducti

Governance is where a policy stops being a document.

Scoped to organization, team, and key, enforced before provider dispatch, and recorded in an audit trail that cannot be edited — that is the difference between having AI policy and having it run.

The model: https://nrouter.ai/ai-governance

Disclosure: I work with the nRouter team.

Production connection pools fail in four predictable ways — and none of them show up in staging:

1. GOAWAY ambush: providers terminate connections on deploy; naive proxies keep dispatching onto the dying socket.
2. Silent idle close: undocumented provider idle timeouts (about 45–60s) reset sockets you still think are warm.
3. Stream-limit cliff: exceed SETTINGS_MAX_CONCURRENT_STREAMS and the whole session dies, taking every in-flight request with it.
4. Stalled pool: a regional outage hangs requests until transport timeouts fire.

The fixes — drain to last_stream_id, 25-second PING probes with proactive socket retirement, dynamic striping at 75% of the stream limit, and a circuit breaker that re-routes in under 5ms — are documented in nRouter's gateway teardown, with the full benchmark (1,000 requests, 1,200-token prompts, gpt-4o-mini, us-east-1) and gateway overhead under 1ms at p50 / under 2ms at p95: https://nrouter.ai/blog/engine....ering/sub-2ms-llm-ga

Your app talks to one endpoint. Behind it, every request is routed, checked, and logged.

An LLM gateway centralizes provider selection, failover, safety checks, budgets, and cost attribution, so a multi-provider setup stops being four implementations of the same layer.

The full pattern: https://nrouter.ai/llm-gateway

Disclosure: I work with the nRouter team.