Evaluate LLMs and AI Agents

This guide shows how to run ARMS evaluations programmatically from your own code - the same evaluation engine, evaluators used for online/auto evaluations, called directly via the SDK.

Install the SDK

pip install --extra-index-url https://arms-packages.elsaifoundry.ai/root/elsai-arms/ elsai-arms==3.0.3

Get an API key

In ARMS, go to Settings → API Keys and create a key. Programmatic evaluations run against your ARMS server, so you'll need this key plus your ARMS URL.

Run an evaluation

Python

import elsai_arms

elsai_arms.init(
    elsai_arms_url="http://localhost:3000",
    elsai_arms_api_key="elsai-xxxxx",
)

result = elsai_arms.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Lyon.",
    contexts=["Paris is the capital and largest city of France."],
)

assert result.passed, f"Evaluation failed: {result.failed_evals}"

You can also set the URL and API key via ELSAI_ARMS_URL / ELSAI_ARMS_API_KEY environment variables instead of init().

Evaluate a whole dataset

Pass a list of prompt/response pairs to run them concurrently - useful as a CI/CD quality gate:

Python

batch_result = elsai_arms.eval_batch(dataset=[
    {"prompt": "What is 2+2?", "response": "2+2 equals 4."},
    {"prompt": "Who wrote Hamlet?", "response": "Hamlet was written by Charles Dickens."},
])

assert batch_result.all_passed, f"Pass rate: {batch_result.pass_rate:.0%}"

Evaluations run programmatically use the exact same evaluators and custom types configured in your ARMS dashboard - trace attributes like service.name and deployment.environment are auto-resolved from elsai_arms.init(), or you can pass your own via the attributes parameter.