InMemoryEvalStore
AgentPrism.Core.dllA store that keeps eval suites, cases, and runs in process memory.
public sealed class InMemoryEvalStore : IEvalStoreInheritance
Section titled “Inheritance”Implements
Section titled “Implements”Inherited Members
Section titled “Inherited Members”object.GetType(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()
Remarks
Section titled “Remarks”Limits: process lifetime and a single node. Use
AgentPrism.PostgreSql in production.
Constructors
Section titled “Constructors”InMemoryEvalStore()
Section titled “ InMemoryEvalStore()”public InMemoryEvalStore()Methods
Section titled “Methods”AddCaseAsync(Guid, EvalCaseDraft, CancellationToken)
Section titled “ AddCaseAsync(Guid, EvalCaseDraft, CancellationToken)”Adds a SINGLE case to the suite. Seq is generated atomically by
the store; the caller does not compute it. If the same
EvalCaseDraft.SourceRunId is added a second time, the
existing case is returned with Created: false.
public ValueTask<EvalCaseAddResult> AddCaseAsync(Guid suiteId, EvalCaseDraft draft, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
draft EvalCaseDraft
The draft of the case to add.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The added (or already existing) case, and whether it was created.
CompleteRunAsync(EvalRunCompletion, CancellationToken)
Section titled “ CompleteRunAsync(EvalRunCompletion, CancellationToken)”Finalizes a run and writes its summary counters.
public ValueTask CompleteRunAsync(EvalRunCompletion completion, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”completion EvalRunCompletion
The finalization information.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
CreateRunAsync(EvalRun, CancellationToken)
Section titled “ CreateRunAsync(EvalRun, CancellationToken)”Creates a new run record (state EvalRunStatus.Pending).
public ValueTask<EvalRun> CreateRunAsync(EvalRun run, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”run EvalRun
The run record.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The run record with its identifier assigned.
DeleteSuiteAsync(string, string, CancellationToken)
Section titled “ DeleteSuiteAsync(string, string, CancellationToken)”Deletes the suite and all of its cases/runs (cascade).
public ValueTask<bool> DeleteSuiteAsync(string tenantId, string name, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
name string
The suite name.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”true if the suite was deleted.
GetRunAsync(string, Guid, CancellationToken)
Section titled “ GetRunAsync(string, Guid, CancellationToken)”Fetches a single run record.
public ValueTask<EvalRun?> GetRunAsync(string tenantId, Guid evalRunId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
evalRunId Guid
The run identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The record; null if it does not exist or belongs to another tenant.
GetRunByJobIdAsync(string, Guid, CancellationToken)
Section titled “ GetRunByJobIdAsync(string, Guid, CancellationToken)”Fetches the run a job record produced.
public ValueTask<EvalRun?> GetRunByJobIdAsync(string tenantId, Guid jobId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
jobId Guid
The job identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The record; null if it does not exist.
GetSuiteAsync(string, string, CancellationToken)
Section titled “ GetSuiteAsync(string, string, CancellationToken)”Fetches the suite matching the given name within the tenant.
public ValueTask<EvalSuite?> GetSuiteAsync(string tenantId, string name, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
name string
The suite name.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The suite; null if it does not exist.
ListCaseResultsAsync(string, Guid, CancellationToken)
Section titled “ ListCaseResultsAsync(string, Guid, CancellationToken)”Lists a run’s case results.
public ValueTask<IReadOnlyList<EvalCaseResult>> ListCaseResultsAsync(string tenantId, Guid evalRunId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
evalRunId Guid
The run identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCaseResult>>
The results.
ListCasesAsync(Guid, CancellationToken)
Section titled “ ListCasesAsync(Guid, CancellationToken)”Lists a suite’s cases, ordered by sequence number.
public ValueTask<IReadOnlyList<EvalCase>> ListCasesAsync(Guid suiteId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCase>>
The cases.
ListSuitesAsync(string, CancellationToken)
Section titled “ ListSuitesAsync(string, CancellationToken)”Lists all of a tenant’s suites.
public ValueTask<IReadOnlyList<EvalSuite>> ListSuitesAsync(string tenantId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalSuite>>
The suites, ordered by name.
MarkRunRunningAsync(Guid, int?, string?, CancellationToken)
Section titled “ MarkRunRunningAsync(Guid, int?, string?, CancellationToken)”Transitions a run to EvalRunStatus.Running and records the agent version and model identifier being measured.
public ValueTask MarkRunRunningAsync(Guid evalRunId, int? agentVersion, string? modelId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”evalRunId Guid
The run identifier.
agentVersion int?
The agent definition version being measured.
modelId string?
The model identifier being measured.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
QueryRunsAsync(EvalRunQuery, CancellationToken)
Section titled “ QueryRunsAsync(EvalRunQuery, CancellationToken)”Lists runs by filter. The newest record is returned first.
public ValueTask<IReadOnlyList<EvalRun>> QueryRunsAsync(EvalRunQuery query, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”query EvalRunQuery
The filter.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalRun>>
The records.
RecordCaseResultAsync(EvalCaseResult, CancellationToken)
Section titled “ RecordCaseResultAsync(EvalCaseResult, CancellationToken)”Records a case result.
public ValueTask RecordCaseResultAsync(EvalCaseResult result, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”result EvalCaseResult
The case result.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The completion task.
ReplaceCasesAsync(Guid, IReadOnlyList<EvalCase>, CancellationToken)
Section titled “ ReplaceCasesAsync(Guid, IReadOnlyList<EvalCase>, CancellationToken)”Replaces all of a suite’s cases with the given list.
public ValueTask<IReadOnlyList<EvalCase>> ReplaceCasesAsync(Guid suiteId, IReadOnlyList<EvalCase> cases, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suiteId Guid
The suite identifier.
cases IReadOnlyList<EvalCase>
The new case list. Sequence numbers are assigned by list order.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<IReadOnlyList<EvalCase>>
The cases with identifiers assigned.
SaveSuiteAsync(EvalSuite, CancellationToken)
Section titled “ SaveSuiteAsync(EvalSuite, CancellationToken)”Creates or updates the suite.
public ValueTask<EvalSuite> SaveSuiteAsync(EvalSuite suite, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”suite EvalSuite
The suite to save.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”The suite with its identifier and timestamps assigned.