Appearance
LLM Gateway
elsai LLM Gateway is a policy and cost-control layer. Put it between your applications and your model providers to decide which models each team may use, cap spend, route requests to the right model automatically, cache responses, and block risky prompts, all from two YAML files.
It runs in two ways:
- As a server (
LLMGateway), an OpenAI-compatible endpoint that your applications call with a virtual key. - In-process (
Gateway), the same policies, budgets and routing aroundlitellm.Router, with no server to run.
Why elsai LLM Gateway?
Key Features
Policies
Restrict models, providers, regions and max_tokens per default, domain, organisation, team and key. The most restrictive result across scopes always wins.
Budgets
Daily or monthly USD limits per scope, with alert thresholds. When a budget is used up, block the request or downgrade to a cheaper model.
Auto Routing
Send model: "auto" and let rules pick the model by task type, prompt size, images, tools, automatically scored prompt complexity, or lowest estimated cost.
Guardrail Templates
Switch on ready-made bundles for PII masking, prompt injection, secrets, harmful content, SQL injection and more with a single line in policy.yaml.
Response Caching
Exact-match caching with per-tenant namespaces and policy-controlled TTLs, so one team never reads another team's cached answer.
Admin API
Manage organisations, domains, teams, keys, budgets and routing rules over /admin/v1, with an audit trail of every change.
Secure by Default
A deny-by-default edge exposes only the data-plane, admin and health routes. Provider keys and admin keys can come from AWS Secrets Manager.
How it works
Every request passes through one policy and budget hook before it reaches a provider:
- Identify the caller from its virtual key (team, key, domain, user).
- Resolve
model: "auto"to a concrete model using routing rules. - Check policy: model, provider, region,
max_tokens, external-provider rules. - Check budget: block, or downgrade to a cheaper model.
- Apply guardrails and the cache, then call the provider with fallbacks.
- Record spend and raise alerts at the configured thresholds.