Python SDK

The SDK runs the gateway's policies, budgets, routing and cache around litellm.Router in your own process, with no proxy server. Importing the package does not import LiteLLM or start anything.

from elsai_llm_gateway import Gateway

gw = Gateway(config="config.yaml", policy="policy.yaml")
resp = gw.completion(
    model="auto",
    messages=[{"role": "user", "content": "hi"}],
    team="payments",
    user="u123",
)
print(resp.choices[0].message.content)
print(resp._hidden_params["gateway"])   # what the gateway decided
gw.close()

The response is the usual OpenAI-shaped ModelResponse.

Module-level functions

Like litellm.completion, with a default gateway read from GATEWAY_CONFIG and GATEWAY_POLICY:

from elsai_llm_gateway import completion, acompletion

resp = completion(model="azure-openai", messages=[{"role": "user", "content": "hi"}])

acompletion is the async version.

Identifying the caller

Pass team, key, org and domain as keyword arguments so the matching policies and budgets apply. Pass user to count spend per user.

Handling refusals

from elsai_llm_gateway import Gateway, PolicyViolation, BudgetExceeded, GatewayDenied

try:
    gw.completion(model="bedrock", messages=msgs, team="research-team")
except PolicyViolation as e:
    print(e.status_code, e.code, e.message)     # 403 policy_violation ...
except BudgetExceeded as e:
    print(e.status_code, e.code)                # 402 budget_exceeded

Exception	Meaning
GatewayDenied	Base class. Has status_code, code and message, matching the HTTP gateway.
PolicyViolation	Model not allowed, token limit, unknown domain, no route, and so on.
BudgetExceeded	A budget is used up and has no downgrade.
GatewayError	Gateway could not start (for example, a bad config).

Differences from the server

• Policies and budgets come from policy.yaml only; there are no database-backed overrides and no admin API.
• Guardrail templates are not applied.
• Spend counters are in memory. For a budget shared between processes, pass a Redis-backed cache:

gw = Gateway(config="config.yaml", policy="policy.yaml", cache=my_redis_cache)

Running the server from Python

from elsai_llm_gateway import LLMGateway

LLMGateway(config="config.yaml", policy="policy.yaml").run(host="127.0.0.1", port=4000)

LITELLM_MASTER_KEY and GATEWAY_ADMIN_KEY are read from the environment. Raises FileNotFoundError if a file does not exist.