Run Evaluations

Start, monitor, and retrieve trust Evaluations programmatically using the CLI, REST API, or MCP.

Vijil Evaluate is a quality assurance framework that automates the testing of LLM applications. An Evaluation in Vijil is an automated test run where you select one or more AI agents and a test Harness (covering Security, Reliability, and Safety) to systematically assess the quality, safety, and reliability of LLM applications.

Prerequisites

• CLI configured and authenticated, or a Bearer token from POST /auth/jwt/login, see the Quickstart
• A registered Agent and its UUID

Start an Evaluation

CLI

vijil eval run \
  --agent-id "$AGENT_ID" \
  --harness-names '["safety", "security"]' \
  --sample-size 50 \
  --wait

Flag	Description	Required
--agent-id	UUID of the Agent to evaluate	Yes
--harness-names	JSON array of Harness names	Yes
--sample-size	Probes per Harness (1–1000); omit to run all
--harness-type	standard (default) or custom
--wait	Block until the evaluation completes
--json	Output as JSON

Use --sample-size 10 for fast iteration during development. Run the full Harness before releasing to production.

MCP

With the Vijil MCP server configured, ask Claude Code in natural language:

Run a safety and security evaluation on agent a1b2c3d4-... with sample size 50, and wait for it to complete

CopiedCopy prompt

Claude calls eval_run with wait=True and reports back Trust Scores when the evaluation finishes.

REST API

The API returns 202 Accepted immediately. Use the returned evaluation_id to poll for status.

curl -s -X POST "$VIJIL_URL/evaluations/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"agent_id\": \"$AGENT_ID\",
    \"team_id\": \"$TEAM_ID\",
    \"harness_names\": [\"safety\", \"security\"],
    \"sample_size\": 50
  }"

Response:

{
  "evaluation_id": "e5f6a7b8-...",
  "status": "starting"
}

Save the evaluation ID:

export EVAL_ID="e5f6a7b8-..."

Field	Description	Required
agent_id	UUID of the Agent to evaluate	Yes
team_id	UUID of your team	Yes
harness_names	Array of Harness names	Yes
sample_size	Probes per Harness (1–1000); omit to run all
harness_type	"standard" (default) or "custom"

Check Status

Status progresses through: starting → pending → running → completed → saving → saved. It may also be failed or canceled.

CLI

vijil eval status <evaluation_id>

curl -s "$VIJIL_URL/evaluations/$EVAL_ID" \
  -H "Authorization: Bearer $TOKEN"

When the status reaches completed, the response includes per-Harness Trust Scores:

{
  "evaluation_id": "e5f6a7b8-...",
  "status": "completed",
  "scores": {
    "safety": 0.82,
    "security": 0.67
  },
  "completed_at": 1712506200
}

Retrieve Results

Get the full per-Probe breakdown once the evaluation status is saved.

CLI

vijil eval results-detail <evaluation_id> --json | jq '.scores'

curl -s "$VIJIL_URL/evaluation-results/$EVAL_ID/results?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN"

Generate a Report

Produce a Trust Report for sharing or archiving.

CLI

vijil eval report <evaluation_id>

curl -s "$VIJIL_URL/evaluations/$EVAL_ID/html?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -o report.html

curl -s "$VIJIL_URL/evaluations/$EVAL_ID/pdf?team_id=$TEAM_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -o report.pdf

List and Cancel Evaluations

List

vijil eval list --agent-id "$AGENT_ID" --status completed

vijil eval cancel <evaluation_id>

vijil eval delete <evaluation_id>

Use GET /evaluations/ and POST /evaluations/{evaluation_id}/cancel for the REST API equivalents.

Next Steps

Understand Results

Deep dive into scores and failures

Custom Harnesses

Create targeted Evaluations

Cloud Providers

Configure provider integrations