Overview

The ARMS SDK is Python only. Offline evaluations use elsai_arms.eval(), elsai_arms.eval_batch(), and elsai_arms.get_eval_types(). There is no JavaScript or TypeScript elsai_arms package.

Evaluations use the same engine, rules, contexts, and custom eval types configured in the ARMS dashboard — for development (offline) and production (online).

Offline Evaluations

Offline evaluations run on the ARMS server using the same evaluation engine as online/auto evaluations. The SDK sends your prompt and response to the server, which runs LLM-as-judge evaluation and returns structured results.

Prerequisites

1. Install the SDK:

pip install --extra-index-url https://arms-packages.elsaifoundry.ai/root/elsai-arms/ elsai-arms==3.0.3

2. A running ARMS instance with evaluation configured in the dashboard.
3. An ARMS API key (create one in the dashboard under Settings > API Keys).

Quick Start

# Option 1: Configure once via init()
elsai_arms.init(
    elsai_arms_url="http://localhost:3000",
    elsai_arms_api_key="elsai-xxxxx",
)

# Run evaluation
result = elsai_arms.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Lyon.",
    contexts=["Paris is the capital and largest city of France."],
)

# Use in assertions
assert result.passed, f"Evaluation failed: {result.failed_evals}"

# Option 2: Pass credentials directly (overrides init/env vars)
result = elsai_arms.eval(
    prompt="Explain quantum computing",
    response="Quantum computers use qubits...",
    elsai_arms_url="http://localhost:3000",
    elsai_arms_api_key="elsai-xxxxx",
)

You can also configure via environment variables:

export ELSAI_ARMS_URL="http://localhost:3000"
export ELSAI_ARMS_API_KEY="elsai-xxxxx"

elsai_arms.eval() Parameters

Parameter	Type	Description	Default
prompt	str	The user prompt sent to the LLM. Required.	-
response	str	The LLM's response to evaluate. Required.	-
contexts	list[str]	Ground truth context for the evaluation.	None
eval_types	list[str]	Specific eval types to run (e.g. ["hallucination", "toxicity"]). Runs all enabled types if omitted.	None
attributes	dict	Trace attributes for rule engine matching (overrides auto-resolved attributes).	None
threshold_score	float	Score threshold for verdict determination.	0.5
store_results	bool	Whether to store results in the ARMS database.	True
run_id	str	Identifier to group related evaluations.	None
metadata	dict	Custom key-value metadata to attach to results.	None
elsai_arms_api_key	str	API key (overrides init() and env var).	None
elsai_arms_url	str	Server URL (overrides init() and env var).	None
print_results	bool	Print formatted summary to terminal.	True

Result Object

elsai_arms.eval() returns an OfflineEvalResult with these properties:

Property	Type	Description
success	bool	Whether the evaluation completed without errors.
passed	bool	True if no evaluation types returned a "yes" verdict.
evaluations	list[OfflineEvaluation]	Individual evaluation results per type.
failed_evals	list[OfflineEvaluation]	Evaluations that returned a "yes" verdict.
context_applied	ContextInfo	Information about rule-matched context.
metadata	dict	Model, run ID, token usage, and cost metadata.
error	str	Error message if success is False.

Each OfflineEvaluation contains:

Field	Type	Description
type	str	The evaluation type (e.g. "hallucination").
score	float	The evaluation score (0.0 to 1.0).
verdict	str	"yes" if detected, "no" otherwise.
classification	str	Category of the detection or "none".
explanation	str	Brief explanation of the evaluation result.

Selecting Evaluation Types

Run specific evaluation types instead of all enabled ones:

result = elsai_arms.eval(
    prompt="Discuss workplace equality",
    response="Older workers can't learn new tech.",
    eval_types=["bias", "toxicity"],
)

Discover Available Types

Python

types = elsai_arms.get_eval_types()
for t in types:
    print(f"{t.id}: {t.label} (custom={t.is_custom}, enabled={t.enabled})")

Batch Evaluation

Evaluate multiple prompt/response pairs concurrently:

Python

dataset = [
    {
        "prompt": "What is 2+2?",
        "response": "2+2 equals 4.",
        "contexts": ["Basic arithmetic."],
    },
    {
        "prompt": "Who wrote Hamlet?",
        "response": "Hamlet was written by Charles Dickens.",
    },
    {
        "prompt": "Describe gravity",
        "response": "Gravity is the force of attraction between masses.",
        "eval_types": ["hallucination"],
    },
]

batch_result = elsai_arms.eval_batch(
    dataset=dataset,
    eval_types=["hallucination", "toxicity"],
    max_concurrent=5,
)

print(f"Pass rate: {batch_result.pass_rate:.0%}")
assert batch_result.all_passed

elsai_arms.eval_batch() Parameters

Parameter	Type	Description	Default
dataset	list[dict]	List of items with prompt and response keys. Required.	-
eval_types	list[str]	Default eval types (can be overridden per item).	None
attributes	dict	Default attributes (can be overridden per item).	None
threshold_score	float	Default threshold score.	0.5
store_results	bool	Store all results in the database.	True
run_id	str	Group all batch evaluations under this ID. Auto-generated if omitted.	None
max_concurrent	int	Maximum number of concurrent evaluations.	5
print_results	bool	Print aggregate summary to terminal.	True

Automatic Attribute Resolution

The SDK automatically resolves trace attributes for rule engine matching, enabling context-aware evaluations without extra configuration. The resolution order (last wins):

1. OTEL_RESOURCE_ATTRIBUTES environment variable
2. OTEL_SERVICE_NAME environment variable
3. ELSAI_ARMS_ENVIRONMENT / OTEL_DEPLOYMENT_ENVIRONMENT environment variable
4. elsai_arms.init() configuration (application_name, environment)
5. Explicit attributes parameter (highest priority)

Python

# These are auto-detected for rule matching:
elsai_arms.init(
    application_name="my-chatbot",
    environment="staging",
)

# Rules configured in the dashboard for service.name="my-chatbot"
# and deployment.environment="staging" will automatically match.
result = elsai_arms.eval(
    prompt="Hello",
    response="Hi there!",
)

# Override auto-resolved attributes:
result = elsai_arms.eval(
    prompt="Hello",
    response="Hi there!",
    attributes={
        "service.name": "different-service",
        "custom.tag": "experiment-v2",
    },
)

CI/CD Integration

Use offline evaluations in your test suite or CI pipeline:

Python

def test_no_hallucination():
    result = elsai_arms.eval(
        prompt="What year did WW2 end?",
        response="World War 2 ended in 1945.",
        eval_types=["hallucination"],
        print_results=False,
    )
    assert result.passed, f"Hallucination detected: {result.failed_evals}"

def test_batch_quality():
    dataset = load_test_cases()  # your test data
    result = elsai_arms.eval_batch(
        dataset=dataset,
        print_results=False,
    )
    assert result.pass_rate >= 0.95, f"Pass rate too low: {result.pass_rate:.0%}"

Configuration Precedence

For elsai_arms_api_key and elsai_arms_url, the resolution order is:

1. Explicit function parameter (highest priority)
2. elsai_arms.init() configuration
3. ELSAI_ARMS_API_KEY / ELSAI_ARMS_URL environment variables