IEvalStore
AgentPrism.Abstractions.dllThe store for evaluation (eval) suites, cases, and runs.
public interface IEvalStoreMethods
Section titled “Methods”AddCaseAsync(Guid, EvalCaseDraft, CancellationToken)
Section titled “ AddCaseAsync(Guid, EvalCaseDraft, CancellationToken)”Adds a SINGLE case to the suite. Seq is generated atomically by
the store; the caller does not compute it. If the same
EvalCaseDraft.SourceRunId is added a second time, the
existing case is returned with Created: false.
ValueTask<EvalCaseAddResult> AddCaseAsync(Guid suiteId, EvalCaseDraft draft, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
draft EvalCaseDraft
The draft of the case to add.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The added (or already existing) case, and whether it was created.
CompleteRunAsync(EvalRunCompletion, CancellationToken)
Section titled “ CompleteRunAsync(EvalRunCompletion, CancellationToken)”Finalizes a run and writes its summary counters.
ValueTask CompleteRunAsync(EvalRunCompletion completion, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”completion EvalRunCompletion
The finalization information.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
CreateRunAsync(EvalRun, CancellationToken)
Section titled “ CreateRunAsync(EvalRun, CancellationToken)”Creates a new run record (state EvalRunStatus.Pending).
ValueTask<EvalRun> CreateRunAsync(EvalRun run, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”run EvalRun
The run record.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The run record with its identifier assigned.
DeleteSuiteAsync(string, string, CancellationToken)
Section titled “ DeleteSuiteAsync(string, string, CancellationToken)”Deletes the suite and all of its cases/runs (cascade).
ValueTask<bool> DeleteSuiteAsync(string tenantId, string name, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
name string
The suite name.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”true if the suite was deleted.
GetRunAsync(string, Guid, CancellationToken)
Section titled “ GetRunAsync(string, Guid, CancellationToken)”Fetches a single run record.
ValueTask<EvalRun?> GetRunAsync(string tenantId, Guid evalRunId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
evalRunId Guid
The run identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The record; null if it does not exist or belongs to another tenant.
GetRunByJobIdAsync(string, Guid, CancellationToken)
Section titled “ GetRunByJobIdAsync(string, Guid, CancellationToken)”Fetches the run a job record produced.
ValueTask<EvalRun?> GetRunByJobIdAsync(string tenantId, Guid jobId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
jobId Guid
The job identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The record; null if it does not exist.
GetSuiteAsync(string, string, CancellationToken)
Section titled “ GetSuiteAsync(string, string, CancellationToken)”Fetches the suite matching the given name within the tenant.
ValueTask<EvalSuite?> GetSuiteAsync(string tenantId, string name, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
name string
The suite name.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The suite; null if it does not exist.
ListCaseResultsAsync(string, Guid, CancellationToken)
Section titled “ ListCaseResultsAsync(string, Guid, CancellationToken)”Lists a run’s case results.
ValueTask<IReadOnlyList<EvalCaseResult>> ListCaseResultsAsync(string tenantId, Guid evalRunId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
evalRunId Guid
The run identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCaseResult>>
The results.
ListCasesAsync(Guid, CancellationToken)
Section titled “ ListCasesAsync(Guid, CancellationToken)”Lists a suite’s cases, ordered by sequence number.
ValueTask<IReadOnlyList<EvalCase>> ListCasesAsync(Guid suiteId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCase>>
The cases.
ListSuitesAsync(string, CancellationToken)
Section titled “ ListSuitesAsync(string, CancellationToken)”Lists all of a tenant’s suites.
ValueTask<IReadOnlyList<EvalSuite>> ListSuitesAsync(string tenantId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalSuite>>
The suites, ordered by name.
MarkRunRunningAsync(Guid, int?, string?, CancellationToken)
Section titled “ MarkRunRunningAsync(Guid, int?, string?, CancellationToken)”Transitions a run to EvalRunStatus.Running and records the agent version and model identifier being measured.
ValueTask MarkRunRunningAsync(Guid evalRunId, int? agentVersion, string? modelId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”evalRunId Guid
The run identifier.
agentVersion int?
The agent definition version being measured.
modelId string?
The model identifier being measured.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
QueryRunsAsync(EvalRunQuery, CancellationToken)
Section titled “ QueryRunsAsync(EvalRunQuery, CancellationToken)”Lists runs by filter. The newest record is returned first.
ValueTask<IReadOnlyList<EvalRun>> QueryRunsAsync(EvalRunQuery query, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”query EvalRunQuery
The filter.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalRun>>
The records.
RecordCaseResultAsync(EvalCaseResult, CancellationToken)
Section titled “ RecordCaseResultAsync(EvalCaseResult, CancellationToken)”Records a case result.
ValueTask RecordCaseResultAsync(EvalCaseResult result, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”result EvalCaseResult
The case result.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
ReplaceCasesAsync(Guid, IReadOnlyList<EvalCase>, CancellationToken)
Section titled “ ReplaceCasesAsync(Guid, IReadOnlyList<EvalCase>, CancellationToken)”Replaces all of a suite’s cases with the given list.
ValueTask<IReadOnlyList<EvalCase>> ReplaceCasesAsync(Guid suiteId, IReadOnlyList<EvalCase> cases, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
cases IReadOnlyList<EvalCase>
The new case list. Sequence numbers are assigned by list order.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCase>>
The cases with identifiers assigned.
SaveSuiteAsync(EvalSuite, CancellationToken)
Section titled “ SaveSuiteAsync(EvalSuite, CancellationToken)”Creates or updates the suite.
ValueTask<EvalSuite> SaveSuiteAsync(EvalSuite suite, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suite EvalSuite
The suite to save.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The suite with its identifier and timestamps assigned.