A drop-in proxy that sits between every application and every LLM provider you use. PHI redaction, prompt-injection blocking, cost governance, and full agentic containment — enforced on every call, with zero code changes.
Every call passes through four governance layers before it ever reaches an LLM.
Built for teams that need control, not just a proxy.
15 entity types including MRN, NPI, ICD-10, CPT codes, drug names, and provider identities. Full round-trip: redact before the LLM sees it, restore in the response. Session-scoped deterministic tokenization keeps the same entity mapped to the same token across a 50-turn agent run.
35+ regex patterns for jailbreaks, token smuggling, and prompt extraction, plus an optional pluggable ML classifier layer for paraphrased attacks that regex misses.
OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Gemini, Groq, Mistral, DeepSeek, Cohere, Fireworks, Together AI, Ollama, xAI, Perplexity, and more. Switch providers with zero code changes.
Pricing auto-refreshes from a maintained source with a local override for negotiated rates — no more hand-editing a stale price table when a provider ships a new model.
/v1/embeddings now runs through the full governance pipeline, and backs an embedding-aware cache layer that catches true paraphrases, not just near-identical strings — 40%+ cost reduction on repetitive workloads.
Per-team, per-user, per-feature, and per-agent-run cost breakdowns with CSV export and anomaly attribution — see exactly which workflow caused the spend spike.
Discover tool servers your agents are calling that were never registered, block unsanctioned calls automatically, and cut off a runaway agent session or MCP server with one action.
Sub-agent depth limiting, per-run cost caps, infinite-loop detection via fingerprinting, and consistent PHI redaction across every turn of an agent session.
Prometheus metrics, OpenTelemetry traces, and a hash-chained tamper-evident audit log. EMA-based anomaly detection on spend and error-rate spikes, delivered via Slack, MS Teams, PagerDuty, or email.
Round-robin, weighted, least-latency, cost-optimized, quality-scored, and fallback — with a circuit breaker per provider and data-residency enforcement for PHI traffic.
RS256/ES256 JWT validation against Microsoft's JWKS endpoint, App Role enforcement, and multi-tenant support — no separate API keys required for org members.
Versioned prompt templates served by the gateway, with built-in A/B testing and lightweight LLM-as-judge evals reusing the same experiments engine.
Full in-process pipeline overhead, single worker, zero-latency mock provider.
Measured with our in-repo benchmark suite (benchmarks/benchmark_gateway.py). This is pipeline overhead, not a network hop — it scales horizontally rather than chasing a single-node throughput ceiling.
Self-host on your own infrastructure, or let us run it — your data path is the same either way.
HIPAA-grade AI workflows with PHI redaction across 15 entity types. Output guardrails scan LLM responses for PHI leakage before they reach your application.
PII redaction for customer data, budget controls per team and product line, a policy engine to block regulated content, and a full audit trail.
Prevent confidential client data from reaching public LLM APIs. Route sensitive matters to on-premise models while using cloud LLMs for non-sensitive work.
MCP tool-call governance, shadow-server detection, and a per-session kill switch for teams running fleets of autonomous agents against real infrastructure.
Same governance pipeline underneath. Choose who manages the infrastructure.
Watch a request flow through all 13 pipeline stages in real time. PHI gets redacted before the LLM call, then restored in the response — the model never sees real patient data.
Launch Interactive Demo →Free forever on Open Source. Pay for usage on Cloud, or a flat license for on-prem.