Evaluations

Automated AI evaluation to assess and monitor the quality, safety, and performance of your LLM outputs. Open Configuration → Evaluations (/evaluations redirects to /evaluations/settings). Types live at /evaluations/types. For evaluation types and custom evaluators, see Evaluators. There is no Analytics tab.

Find the right feature

If you want to...	Use this ARMS feature
Automatically score every production trace	Configuration (Auto Evaluation)
Score one specific trace on demand	LLM-as-a-Judge from that trace's Evaluation tab
Rate a trace yourself instead of an LLM judge	Manual Feedback - Good / Bad / Neutral plus a comment
Evaluate on criteria beyond the 11 built-in types	Custom evaluators
Test prompt or model changes before shipping	Programmatic evaluations via the SDK
Block a deploy on a quality regression	Programmatic evaluations in a CI/CD pipeline
Inspect evaluation types and enable/disable them	Evaluation Types

Setup

Online

Pick a judge model

Go to Evaluations → Settings, choose a provider and model to act as the judge (OpenAI, Anthropic, Google, Mistral, and 7+ others), and add its API key from Vault.

Enable evaluators

Switch to the Evaluation Types tab and turn on the evaluators you want. Hallucination, Bias, and Toxicity are enabled by default; Relevance, Coherence, Safety, and 5 others are opt-in.

Turn on Auto Evaluation

Back in Settings, enable Auto Evaluation with a cron schedule so every new trace gets scored automatically - or skip this and click Run Evaluation from any trace's Evaluation tab to score it on demand.

Review results

Open any trace's Evaluation tab for its score, classification, and reasoning, or check the Evaluations dashboard for aggregate trends across models and time.

Offline

Install the SDK

pip install --extra-index-url https://arms-packages.elsaifoundry.ai/root/elsai-arms/ elsai-arms==3.0.4

Get an API key

In ARMS, go to Settings → API Keys and create a key. Offline evaluations call your ARMS server, so you'll need this key plus your ARMS URL.

Run an evaluation

import elsai_arms

elsai_arms.init(
    elsai_arms_url="http://localhost:3000",
    elsai_arms_api_key="elsai-xxxxx",
)

result = elsai_arms.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Lyon.",
    contexts=["Paris is the capital and largest city of France."],
)

assert result.passed, f"Evaluation failed: {result.failed_evals}"

The SDK is Python-only. eval, eval_batch, and get_eval_types call /api/evaluation/offline.

This uses the exact same evaluators and custom types configured in your dashboard.