Configuration

The gateway is configured with two files:

File	Defines	Format
config.yaml	Which models exist, provider credentials, retries and fallbacks	LiteLLM config.yaml
policy.yaml	Who may use what: policies, budgets, routing rules, guardrail templates	Policy file

policy.yaml is optional. If you do not pass one, a policy.yaml next to config.yaml is used when present.

config.yaml

model_list:
  - model_name: azure-openai
    litellm_params:
      model: azure/<your-deployment-name>
      api_base: https://<your-resource>.openai.azure.com/
      api_key: os.environ/AZURE_OPENAI_API_KEY
      api_version: 2025-01-01-preview
      region_name: azure              # a label policies can restrict with allowed_regions
    model_info: {id: azure-openai-1}

  - model_name: bedrock
    litellm_params:
      model: bedrock/converse/anthropic.claude-sonnet-4-5-20250929-v1:0
      aws_region_name: us-east-1
    model_info: {id: bedrock-1}

router_settings:
  num_retries: 2
  timeout: 120
  allowed_fails: 3
  cooldown_time: 30
  # If a model fails, the next one is tried. A fallback the caller's policy forbids is dropped.
  fallbacks:
    - azure-openai: [bedrock]
    - bedrock: [azure-openai]

Azure model names

For Azure OpenAI, model must include the deployment name: azure/<deployment>. A bare model: azure is rejected at start-up with GetLLMProvider Exception.

Keep secrets out of the file

Prefer os.environ/NAME over literal keys, and load the values from the environment or AWS Secrets Manager. If you do write literal keys, treat the file as a secret and never commit it.

Values from the environment

Inside any string value you can use ${NAME} or ${NAME:-default}. The gateway fills them in at start-up and runs LiteLLM on a rendered private copy:

model: azure/${AZURE_OPENAI_DEPLOYMENT}
api_base: ${AZURE_OPENAI_ENDPOINT}
region_name: ${AZURE_OPENAI_REGION:-azure}
api_key: os.environ/AZURE_OPENAI_API_KEY

• A variable that is not set and has no default is an error naming the variable and every place that uses it.
• Fields that look like secrets (key, secret, password, token, credential) may not use ${...}; use os.environ/NAME.

policy.yaml

A minimal policy file:

default:
  max_tokens: 4096

teams:
  research-team:
    allowed_models: [azure-openai]

See Policy File for every section.