AI Gateway
Enterprise AI usage management and cost governance platform. Routes, controls, and observes all Claude Code API traffic across a 7-developer engineering team with $92K/month in AI spend via Google Vertex AI.
- Role
- Sole Engineer — Architecture, Infrastructure, CI/CD, Observability
- Timeline
- 3 iterations
Project Overview
The engineering team was using Claude Code (Anthropic's AI coding assistant) via Google Vertex AI. Seven developers across two teams, each with their own service accounts, spending a combined $92K/month — with no centralized budget controls, no model governance, and no per-developer usage tracking. The only visibility into spending was the monthly cloud bill.
I designed and built an AI Gateway — a five-layer platform that sits between developers and the AI provider. Every API request flows through the gateway, which authenticates the developer, enforces budget and model policies, proxies the request to Vertex AI, and records token-level telemetry. The infrastructure runs as 8 Docker Compose services with full observability — 7 Grafana dashboards, Prometheus metrics, Loki logs, and automated alerting via Alertmanager.
The Business Problem
Before the gateway: no per-developer cost attribution, no budget limits or enforcement, no model governance (anyone uses any model), no usage telemetry or audit trail, no centralized management interface, and monthly bills as the only monitoring.
After the gateway: per-developer cost tracking in real-time, daily and monthly budget enforcement with automatic blocking, model allowlists per developer and team, full token-level telemetry and session tracking, a management portal with dashboards and controls, and automated alerts with an emergency kill switch.
Technical Challenges
Three architecture iterations to find the right design
V1 attempted a Cloud Run HTTP gateway using a custom base URL — failed because Claude Code did not support it. V2 used a Squid CONNECT proxy — worked but could not see cost inside TLS tunnels. V3 discovered ANTHROPIC_BASE_URL, enabling an application-level API proxy with direct token visibility. Each failure informed the next design.
Real-time token counting from SSE streams
Claude Code uses Server-Sent Events for streaming responses. The gateway needs to count tokens and compute cost without buffering the entire response. I built an SSE streaming tap that reads each event as it passes through, extracts token usage from the final event, and records cost — all without adding latency to the developer experience.
Per-developer identity across proxy and API layers
The system uses two proxy modes: Squid CONNECT tunnel for legacy clients and FastAPI API proxy for Claude Code. Both need consistent per-developer identity. API keys map to developer manifests, and every request is attributed to the correct developer regardless of which proxy path it takes.
Budget enforcement without blocking the team
If the policy engine goes down, should all AI access stop? I chose fail-open: if the enforcement system is unreachable, requests are allowed. A health cache prevents request storms. Stopping all AI access because of a monitoring failure causes more damage than temporarily allowing unmonitored usage.
Architecture
The system uses a five-layer architecture. All Claude Code API traffic routes through the gateway. The policy engine intercepts every request, enforces budgets and model governance, proxies to Vertex AI, and records usage telemetry. The data flow is: Developer → TLS + Auth (nginx) → Policy Engine (budget + governance) → Vertex AI (Claude models) → Telemetry (Prometheus + Loki).
The gateway core runs as 8 Docker Compose services: Squid forward proxy (CONNECT tunnel for legacy clients), FastAPI policy engine (budget enforcement, model governance, API proxy), SQLite (API keys, sessions, enforcement state, audit log), YAML manifest watcher (policy-as-code with hot-reload), OTLP Collector (OpenTelemetry pipeline), a reload helper sidecar, nginx (TLS termination, rate limiting, SPA routing), and the management portal.
AI Gateway
Enterprise AI Usage Management & Cost Governance Platform
Clients
Developer Machines
Claude Code CLI VS Code / JetBrains IDE
Management Portal
Dashboards & Controls Budget monitoring
Edge — TLS & Rate Limiting
nginx Reverse Proxy
TLS (Let's Encrypt) · HTTP/2 Rate limiting (30r/s) · SSE passthrough
Gateway Core (Docker Compose)
Squid Forward Proxy
CONNECT tunnel htpasswd auth · Access logging
Policy Engine
FastAPI · Budget enforcement Model governance · API proxy
SQLite
API keys · Sessions Enforcement state
Developer Manifests
YAML policy-as-code Hot-reload · Git-controlled
OTLP Collector
Telemetry pipeline Metrics · Logs · Traces
AI Provider
Google Vertex AI
Claude API (Anthropic via Vertex)
Observability & CI/CD
Monitoring Stack
Prometheus · Grafana Loki · Alertmanager
GitHub Actions CI/CD
Self-hosted runner Auto-deploy on push to main
$92K
Monthly AI Spend
7
Active Developers
8
Docker Services
7
Grafana Dashboards
24/7
Monitoring & Alerting
Edge Layer
nginx reverse proxy handling TLS termination via Let's Encrypt, HTTP/2, rate limiting (30 req/s burst=60), SSE streaming passthrough, and X-Request-ID correlation. Routes /v1/* to the policy engine, /policy/* to the management API, and /portal/* to the SPA.
Policy Engine
FastAPI application (Python 3.12) with 14 API modules and a 10-stage proxy pipeline. Authenticates developers via API keys, enforces daily and monthly budgets, governs model access via allowlists, proxies requests to Vertex AI, and taps SSE streams for real-time token counting and cost attribution.
Data Layer
SQLite for API keys, sessions, enforcement state, and audit logs. YAML manifests define per-developer policies (budget limits, allowed models, routing mode) as code — version-controlled in git with hot-reload via filesystem watching.
Observability Layer
OpenTelemetry pipeline exports metrics to Prometheus and logs to Loki. 7 purpose-built Grafana dashboards cover executive summary, per-developer usage, operations, governance compliance, cost attribution, telemetry pipeline health, and policy engine metrics. Alertmanager handles budget alerts via email and Telegram.
CI/CD Layer
GitHub Actions with a self-hosted runner on the gateway VM. Push-to-main triggers automated testing, Docker image rebuild, deployment, and post-deploy health checks. Zero-downtime deployment.
Solution
The production system runs on a GCP Compute Engine VM with Docker Compose orchestrating 8 containerized services, each with hard memory limits and health checks. nginx handles TLS termination via Let's Encrypt, HTTP/2, rate limiting (30 req/s burst=60), and SSE streaming passthrough. Squid provides a CONNECT tunnel for legacy proxy clients. The FastAPI policy engine authenticates every request, enforces daily ($100) and monthly ($2,000) budget limits, governs model access via allowlists, and proxies traffic to Google Vertex AI.
Developer onboarding is policy-as-code: each developer has a YAML manifest in the git repository defining their team, routing mode, allowed models, and budget limits. Changes go through pull requests. The policy engine watches the filesystem and hot-reloads on changes — adding a new developer means committing one YAML file.
The CI/CD pipeline runs on GitHub Actions with a self-hosted runner on the gateway VM. Push-to-main triggers automated testing, Docker image rebuild, deployment, and post-deploy health checks — achieving zero-downtime deployments. The observability stack includes Prometheus for metrics (15s scrape interval), Loki for centralized logs, 7 purpose-built Grafana dashboards, and Alertmanager for budget alerts via email and Telegram.
Key Engineering Decisions
Application-level proxy over network-level proxy
Context
V2 used a Squid CONNECT tunnel — it worked but could only see encrypted bytes, not token counts or model names. Cost attribution required a separate billing pipeline with inherent delay. When ANTHROPIC_BASE_URL was discovered, it opened the door to an application-level proxy.
Outcome
V3 uses ANTHROPIC_BASE_URL to route Claude Code traffic through a FastAPI proxy that can read every request and response in plaintext. This gives direct access to model names, token counts, and cost — no separate billing pipeline needed. Real-time cost attribution instead of delayed reconciliation.
Policy-as-Code with YAML manifests
Context
Developer policies (budgets, model access, routing) could be stored in a database and managed through the portal UI. But policies are infrastructure decisions — they should be version-controlled, reviewable, and auditable.
Outcome
Each developer has a YAML manifest in the git repository. Changes go through pull requests. The policy engine watches the filesystem and hot-reloads on changes. The portal provides read access and operational controls, but the source of truth is git.
Fail-open budget enforcement
Context
The enforcement loop checks budget utilization every 60 seconds. If the policy engine crashes, the system needs to decide: block all traffic (fail-safe) or allow all traffic (fail-open).
Outcome
Fail-open with health caching. Blocking all AI access because of a monitoring failure is worse than temporarily unmonitored usage. Alertmanager notifies the team immediately if enforcement goes stale for more than 5 minutes.
SSE streaming tap instead of response buffering
Context
To count tokens and compute cost, the gateway needs to read the API response. Buffering the entire response would add latency and break the streaming experience developers expect from Claude Code.
Outcome
The proxy taps the SSE stream as it passes through — reading each event without buffering. Token usage is extracted from the final event in the stream. Developers see no added latency. Cost is recorded before the response completes.
Technology Stack
Infrastructure & Orchestration
Docker Compose (8 services), nginx (TLS, rate limiting, SSE passthrough), Squid (CONNECT tunnel), Let's Encrypt, GCP Compute Engine
Observability & Monitoring
Prometheus (15s scrape), Grafana (7 dashboards), Loki, Alertmanager, OpenTelemetry Collector
CI/CD & Automation
GitHub Actions, self-hosted runner, zero-downtime deployment, automated health checks, Docker image rebuild
Policy & Governance
FastAPI policy engine (Python 3.12), YAML manifests (policy-as-code), SQLite, budget enforcement loop (60s), hot-reload via watchfiles
Cloud & AI Provider
Google Vertex AI, 6 Claude models (Opus/Sonnet/Haiku), SSE streaming proxy, per-developer service accounts
Results
$92K
Monthly AI spend under governance
7
Developers managed across 2 teams
6
Claude models governed
8
Docker Compose services in production
7
Purpose-built Grafana dashboards
24/7
Monitoring with automated alerts
The platform provides complete visibility into AI spending — per developer, per model, per day. Budget enforcement runs every 60 seconds. When a developer exceeds their daily or monthly limit, they are automatically blocked within one enforcement cycle and an alert is sent via Alertmanager.
The system evolved through three architecture iterations, each solving limitations discovered in production. The final design was chosen for direct token-level observability and transparent Claude Code integration — developers set one environment variable and the gateway handles everything else.
Lessons Learned
Assumptions kill architectures
V1 was designed around an environment variable that did not exist. The entire architecture had to be scrapped. After this failure, I adopted a strict rule: no architecture decisions based on assumptions — everything requires a working proof of concept first.
Iterate toward the right abstraction level
V1 (HTTP proxy) failed. V2 (network-level CONNECT proxy) worked but had fundamental visibility limitations. V3 (application-level API proxy) solved everything. Each iteration taught me something the previous one could not. The right architecture was not obvious from the start — it emerged through real failure.
Policy-as-code scales better than UI-driven config
Storing developer policies in YAML manifests instead of a database means changes go through pull requests, have git history, and are reviewable. When onboarding a new developer, the team creates one file — not 15 clicks through a portal.
Fail-open is a valid engineering choice
The natural instinct is fail-safe — if something breaks, shut everything down. But for a governance platform supporting a development team, blocking all AI access due to a monitoring failure causes more damage than temporarily allowing unmonitored usage.