Security and Deployment

Deny-by-default edge

The stock LiteLLM proxy serves its whole management surface (keys, teams, users, spend, config, UI, SSO, pass-through routes, files, batches) on the same port as the data plane. Anyone with the master key could change keys or budgets without an audit trail, and pass-through routes would forward raw requests without the policy and budget hook.

The gateway answers 404 to everything outside an allowlist:

Group	Routes
Data plane	/v1/invoke, chat completions, completions, embeddings, images, audio, moderations, rerank, responses, messages, models. All pass through the policy and budget hook.
Admin API	/admin/v1/...
Probes	/health/liveliness, /health/liveness, /health/readiness
Opt-in	/docs, /redoc, /openapi.json (GATEWAY_ENABLE_DOCS=true), /metrics (GATEWAY_ENABLE_METRICS=true), extra prefixes (GATEWAY_EXTRA_ALLOWED_PREFIXES=/a,/b)

GATEWAY_LOCKDOWN=off disables the edge. Use it for development only.

Secrets

• Keep provider keys out of config.yaml; use os.environ/NAME and load them from the environment or AWS Secrets Manager.
• Rendered config copies are written to a private temporary directory and never contain os.environ/ secrets.
• The gateway never logs secret values.
• If a config file with literal keys has ever been shared or committed, rotate those keys.

Network

LLMGateway.run() binds to 0.0.0.0 by default. For local testing, bind to localhost:

LLMGateway(config="config.yaml", policy="policy.yaml").run(host="127.0.0.1", port=4000)

In production, put the gateway behind your own TLS terminator or load balancer.

Start-up checks

elsai-llm-gateway (or python -m elsai_llm_gateway.bootstrap) loads secrets, verifies LITELLM_MASTER_KEY and GATEWAY_ADMIN_KEY, renders the config and starts LiteLLM. If a required value is missing it stops and names it.

Multiple instances

Spend counters and cache must be shared for budgets to hold across instances. Configure Redis in config.yaml and set REDIS_PASSWORD if needed. Use PostgreSQL for virtual keys, teams and admin-managed policies.