OnlineEvalJobHandler
AgentPrism.Core.dllExecutes JobKind.OnlineEval jobs: scores a sampled production run with every registered IRunJudge.
public sealed class OnlineEvalJobHandler : IJobHandlerInheritance
Section titled “Inheritance”Implements
Section titled “Implements”Inherited Members
Section titled “Inherited Members”object.GetType(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()
Remarks
Section titled “Remarks”A judge’s failure does not affect the run being scored. The run has already completed; this job runs in the background. This rule is a direct application of the principle that observability must not break functionality.
The score row’s RunScore.Author field is INTENTIONALLY filled
with judge:{name} (not null like a human score):
this way the run_scores table’s (tenant_id, run_id, message_id, author)
uniqueness constraint kicks in, and retrying this job or manually
repeating it via POST /api/runs/{id}/judge produces an update of the
existing row for the SAME judge, not a second row.
OnlineEvalJobHandler.JudgeRunAsync is the shared core used by BOTH the queued job
(OnlineEvalJobHandler.ExecuteAsync) AND the manual scoring endpoint
(POST /api/runs/{id}/judge); it SKIPS sampling and expects a
RunRecord the caller has already resolved.
Constructors
Section titled “Constructors”OnlineEvalJobHandler(IRunStore, IRunInputStore, IRunScoreStore, IEnumerable<IRunJudge>, ITenantContext, AgentPrismMetrics?, OnlineEvalSummaryService?, TimeProvider?, ILogger<OnlineEvalJobHandler>?)
Section titled “ OnlineEvalJobHandler(IRunStore, IRunInputStore, IRunScoreStore, IEnumerable<IRunJudge>, ITenantContext, AgentPrismMetrics?, OnlineEvalSummaryService?, TimeProvider?, ILogger<OnlineEvalJobHandler>?)”Executes JobKind.OnlineEval jobs: scores a sampled production run with every registered IRunJudge.
public OnlineEvalJobHandler(IRunStore runStore, IRunInputStore runInputStore, IRunScoreStore scoreStore, IEnumerable<IRunJudge> judges, ITenantContext tenantContext, AgentPrismMetrics? metrics = null, OnlineEvalSummaryService? summaryService = null, TimeProvider? timeProvider = null, ILogger<OnlineEvalJobHandler>? logger = null)Parameters
Section titled “Parameters”runStore IRunStore
runInputStore IRunInputStore
scoreStore IRunScoreStore
judges IEnumerable<IRunJudge>
tenantContext ITenantContext
metrics AgentPrismMetrics?
summaryService OnlineEvalSummaryService?
timeProvider TimeProvider?
logger ILogger<OnlineEvalJobHandler>?
Remarks
Section titled “Remarks”A judge’s failure does not affect the run being scored. The run has already completed; this job runs in the background. This rule is a direct application of the principle that observability must not break functionality.
The score row’s RunScore.Author field is INTENTIONALLY filled
with judge:{name} (not null like a human score):
this way the run_scores table’s (tenant_id, run_id, message_id, author)
uniqueness constraint kicks in, and retrying this job or manually
repeating it via POST /api/runs/{id}/judge produces an update of the
existing row for the SAME judge, not a second row.
OnlineEvalJobHandler.JudgeRunAsync is the shared core used by BOTH the queued job
(OnlineEvalJobHandler.ExecuteAsync) AND the manual scoring endpoint
(POST /api/runs/{id}/judge); it SKIPS sampling and expects a
RunRecord the caller has already resolved.
Properties
Section titled “Properties”The job kind this handler can execute.
public JobKind Kind { get; }Property Value
Section titled “Property Value”Methods
Section titled “Methods”ExecuteAsync(JobContext, CancellationToken)
Section titled “ ExecuteAsync(JobContext, CancellationToken)”Executes the job.
public ValueTask ExecuteAsync(JobContext context, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”context JobContext
The job context: record, items, reporting, and cancellation check.
cancellationToken CancellationToken
The cancellation token (triggered when the worker shuts down).
Returns
Section titled “Returns”The completion task.
Remarks
Section titled “Remarks”If the handler throws, the job is retried with IJobStore.ReleaseForRetryAsync (if the attempt limit is not exceeded) or marked as JobStatus.Failed.
JudgeRunAsync(RunRecord, CancellationToken)
Section titled “ JudgeRunAsync(RunRecord, CancellationToken)”Scores the given run with every registered IRunJudge.
public ValueTask<(IReadOnlyList<RunScore> Scores, IReadOnlyList<string> Failures)> JudgeRunAsync(RunRecord run, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”run RunRecord
The run to score. The caller must have already performed the tenant/entity check.
cancellationToken CancellationToken
Cancellation token.
Returns
Section titled “Returns”ValueTask<(IReadOnlyList<RunScore> Scores, IReadOnlyList<string> Failures)>
The scores written, and the name/message of any judges that failed.
Both are empty (not an error) when there is no run_inputs record,
no registered judge, or the output cannot be read.