Evals
13 operations. {prefix} is the route prefix passed to
MapAgentPrism; the template uses /agentprism.
GET {prefix}/api/evals
Section titled “GET {prefix}/api/evals”Operation ID: AgentPrismListEvalSuites
Lists a tenant’s eval suites.
Each entry is a suite’s definition — the agent under test and its check definitions — without the cases or the past runs; read those from the cases and runs endpoints. The response is not paged.
Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → array of EvalSuite |
— |
GET {prefix}/api/evals/{name}
Section titled “GET {prefix}/api/evals/{name}”Operation ID: AgentPrismGetEvalSuite
Gets a single eval suite.
The suite carries its checks, which are stored together with it rather than as separate rows, because a suite’s checks are always read and written as one unit. Cases and runs are separate endpoints. Suites are scoped to the calling tenant, and an unknown name returns 404.
Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → EvalSuite |
— |
PUT {prefix}/api/evals/{name}
Section titled “PUT {prefix}/api/evals/{name}”Operation ID: AgentPrismSaveEvalSuite
Creates or updates an eval suite.
Check definitions are declarative; an unknown check type turns into an error at run time.
Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
Request body (required):
application/json→EvalSuiteSaveRequest
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → EvalSuite |
— |
DELETE {prefix}/api/evals/{name}
Section titled “DELETE {prefix}/api/evals/{name}”Operation ID: AgentPrismDeleteEvalSuite
Deletes an eval suite (together with its cases and runs).
The cases and every past eval run cascade with the suite, so the score history used to compare agent versions disappears with it — export it first if it matters. The agent runs those evals produced stay in the run history and are still readable there. An unknown name returns 404.
Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
| Response | Body | Headers |
|---|---|---|
| 204 No Content | — | — |
GET {prefix}/api/evals/{name}/cases
Section titled “GET {prefix}/api/evals/{name}/cases”Operation ID: AgentPrismListEvalCases
Lists a suite’s cases.
Cases come back in their stored order, and that order is their identity: a case is addressed by its sequence number, so reordering the list changes which case a past result refers to. An unknown suite name returns 404, while a suite with no cases returns an empty list.
Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → array of EvalCase |
— |
PUT {prefix}/api/evals/{name}/cases
Section titled “PUT {prefix}/api/evals/{name}/cases”Operation ID: AgentPrismSaveEvalCases
Replaces all of a suite’s cases with the given list.
This is a full replacement, not an append: cases missing from the body are removed, so send the complete list every time. Sequence numbers are assigned from the body’s order, which means reordering the list re-numbers the cases and past results then line up with different cases. Every case needs a non-empty ‘query’; one that does not fails the whole request with 400 and nothing is written. An unknown suite name returns 404.
Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
Request body (required):
application/json→ array ofEvalCaseInput
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → array of EvalCase |
— |
DELETE {prefix}/api/evals/{name}/cases
Section titled “DELETE {prefix}/api/evals/{name}/cases”Operation ID: AgentPrismClearEvalCases
Deletes all of a suite’s cases.
The suite itself survives with its checks intact; only the cases go. Past eval runs and their per-case results are kept, but they then point at cases that no longer exist. The call is idempotent — clearing an already empty suite still answers 204. An unknown suite name returns 404.
Authorization: bearer authentication; Admin role policy when that policy is registered; EvalsAdmin API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
| Response | Body | Headers |
|---|---|---|
| 204 No Content | — | — |
POST {prefix}/api/evals/{name}/cases/from-run/{runId}
Section titled “POST {prefix}/api/evals/{name}/cases/from-run/{runId}”Operation ID: AgentPrismPromoteRunToEvalCase
Promotes a run to an eval case in a single request.
The query is read from the run’s own session; runs without a session cannot be promoted. If the same run is promoted a second time, the existing case is returned (200, not 201).
Authorization: bearer authentication; Operator role policy when that policy is registered; EvalsAdmin API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
runId |
path | yes | string (uuid) |
— |
Request body (optional):
application/json→EvalCasePromotionRequest
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → EvalCase |
— |
| 201 Created | application/json → EvalCase |
— |
POST {prefix}/api/evals/{name}/run
Section titled “POST {prefix}/api/evals/{name}/run”Operation ID: AgentPrismTriggerEvalRun
Runs an eval suite now.
Each case runs in a new session on the agent being evaluated and produces its own ‘runs’ row. The run is queued as a job; results are processed in the background.
Authorization: bearer authentication; Operator role policy when that policy is registered; RunsWrite API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
Request body (optional):
application/json→EvalRunTriggerRequest
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → EvalRun |
— |
GET {prefix}/api/evals/{name}/runs
Section titled “GET {prefix}/api/evals/{name}/runs”Operation ID: AgentPrismListEvalRuns
Lists a suite’s past runs.
Each entry is one execution of the whole suite with its aggregate outcome; the per-case results live behind the single eval-run endpoint. Comparing entries over time is how a regression between agent versions is spotted. Paging is offset based, with ‘skip’ defaulting to 0 and ‘take’ to 50. An unknown suite name returns 404.
Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
name |
path | yes | string |
— |
skip |
query | no | integer (int32) |
pattern `^-?(?:0\ |
take |
query | no | integer (int32) |
pattern `^-?(?:0\ |
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → array of EvalRun |
— |
GET {prefix}/api/evals/runs/{id}
Section titled “GET {prefix}/api/evals/runs/{id}”Operation ID: AgentPrismGetEvalRun
Gets a single eval run and its per-case results.
This is the endpoint to poll after triggering a suite: the eval run is queued and processed in the background, and its results fill in as cases complete. Each result names the agent run it came from, so a failing check can be traced to the exact conversation. Per-case results are a retention target, so an old eval run may keep its summary while its details are gone. An unknown id, or one belonging to another tenant, returns 404.
Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
id |
path | yes | string (uuid) |
— |
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → EvalRunDetailResponse |
— |
GET {prefix}/api/evaluation/online
Section titled “GET {prefix}/api/evaluation/online”Operation ID: AgentPrismGetOnlineEvaluationSummary
Returns a summary of the online evaluation window.
Returns the average judge score, sample count, and judge cost within the window. The summary is in-memory (it resets when the process restarts); for an authoritative result, the ‘run_scores’ table can be queried directly.
Authorization: bearer authentication; Reader role policy when that policy is registered; EvalsRead API-key scope.
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → OnlineEvaluationSummary |
— |
POST {prefix}/api/runs/{runId}/judge
Section titled “POST {prefix}/api/runs/{runId}/judge”Operation ID: AgentPrismJudgeRun
Manually has judge(s) score a run.
This SKIPS the sampling decision; it is for calibration and debugging. If no IRunJudge is registered, or the run’s input/output cannot be read, an empty list is returned.
Authorization: bearer authentication; Operator role policy when that policy is registered; RunsWrite API-key scope.
| Parameter | In | Required | Type | Description and rules |
|---|---|---|---|---|
runId |
path | yes | string (uuid) |
— |
| Response | Body | Headers |
|---|---|---|
| 200 OK | application/json → array of RunScore |
— |