Quick Start Guide

Get a gateway running in a few minutes.

Step 1: Create config.yaml and policy.yaml

yaml

# config.yaml
model_list:
  - model_name: azure-openai
    litellm_params:
      model: azure/<your-deployment-name>
      api_base: https://<your-resource>.openai.azure.com/
      api_key: os.environ/AZURE_OPENAI_API_KEY
      api_version: 2025-01-01-preview

yaml

# policy.yaml
default:
  max_tokens: 4096

Step 2: Set the required keys

bash

export LITELLM_MASTER_KEY="sk-<random>"
export GATEWAY_ADMIN_KEY="<random>"
export AZURE_OPENAI_API_KEY="<your-provider-key>"

Step 3: Start the server

python

from elsai_llm_gateway import LLMGateway

LLMGateway(config="config.yaml", policy="policy.yaml").run(host="127.0.0.1", port=4000)

Constructing LLMGateway has no side effects; the proxy starts only when you call run().

Check that it is up:

bash

curl http://127.0.0.1:4000/health/liveliness

Step 4: Call it

The gateway is OpenAI compatible:

bash

curl http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "azure-openai",
        "messages": [{"role": "user", "content": "Say hi"}],
        "max_tokens": 20
      }'

Or use the simpler /v1/invoke endpoint, which takes a prompt and returns the answer with the same policies, budgets and routing applied:

bash

curl http://127.0.0.1:4000/v1/invoke \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"prompt": "Summarise the benefits of caching in one sentence."}'

A request that breaks a policy is refused. With the policy above, max_tokens: 5000 returns:

json

{"error": {"message": "max_tokens 5000 exceeds the policy limit of 4096", "type": "invalid_request_error", "code": "400"}}

Step 5: Use it without a server

python

from elsai_llm_gateway import Gateway

gw = Gateway(config="config.yaml", policy="policy.yaml")
resp = gw.completion(
    model="azure-openai",
    messages=[{"role": "user", "content": "Say hi"}],
    team="research-team",
)
print(resp.choices[0].message.content)

Quick Start Guide

Step 1: Create config.yaml and policy.yaml

Step 2: Set the required keys

Step 3: Start the server

Step 4: Call it

Step 5: Use it without a server

Next steps

elsai Docs Assistant

Quick Start Guide ​

Step 1: Create config.yaml and policy.yaml ​

Step 2: Set the required keys ​

Step 3: Start the server ​

Step 4: Call it ​

Step 5: Use it without a server ​

Next steps ​

Quick Start Guide

Step 1: Create config.yaml and policy.yaml

Step 2: Set the required keys

Step 3: Start the server

Step 4: Call it

Step 5: Use it without a server

Next steps