Evaluation Engine

Evaluation Engine is /evaluations/settings. Open it from Configuration → Evaluations (or /evaluations, which redirects here). /settings/evaluation also redirects here. This is a standalone page, not a Configuration tab on an Analytics screen.

Required fields

Field	Notes
Provider	From Manage Models
Model	Models for that provider, or a custom model name
API Key (Vault)	Select a Vault secret, or Create new

Save with Save Changes or Create Config.

Auto Evaluation

Toggle Enable auto evaluation and set a cron schedule (for example */15 * * * *). Auto Evaluation runs enabled evaluators against newly ingested LLM traces.

There is no sample-rate control on this page.

Scheduled jobs require ELSAI_CRON_SECRET on the server. See Configuration.

Which types run is controlled on /evaluations/types. Manual Run Evaluation on a trace uses the same judge.

Manual evaluation

The page explains both Manual and Auto paths. Open a trace in Telemetry and run evaluation from its Evaluations panel, or use Go to Traces. See LLM-as-a-Judge.