ModelRunJudge

Namespace AgentPrism · Assembly AgentPrism.Core.dll

Provides the built-in, model-backed IRunJudge implementation.

public sealed class ModelRunJudge : IRunJudge

Inheritance

object ← ModelRunJudge

Implements

IRunJudge

Inherited Members

object.GetType(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()

Remarks

The judge’s own call goes through a RunRecordingAgent wrapper and is recorded as RunKind.Eval, like every case run from EvalJobHandler. This computes cost automatically and excludes the row from RunStatistics, so the evaluated agent’s cost does not grow and no new column is needed. The recording decorator deliberately disables quota enforcement with quotaEnforcer: null and event publishing with webhookPublisher: null. A judge is a control-plane cost, does not use the user’s quota, and does not publish its own run.completed noise.

Structured output uses the ChatResponseFormat.ForJsonSchema(JsonElement, ...) overload. Its manually written schema does not use reflection and preserves the AOT stance of AgentPrism.Core.

Constructors

ModelRunJudge(IModelProviderRegistry, IOptionsMonitor<ModelRunJudgeOptions>, IRunStore, ITenantContext, IOptions<AgentPrismOptions>, ILoggerFactory, AgentPrismMetrics?, RunTraceCollector?, TimeProvider?, IRunPricingResolver?, IRunErrorClassifier?, IRunInputStore?, IServiceProvider?)

Provides the built-in, model-backed IRunJudge implementation.

public ModelRunJudge(IModelProviderRegistry modelProviders, IOptionsMonitor<ModelRunJudgeOptions> optionsMonitor, IRunStore runStore, ITenantContext tenantContext, IOptions<AgentPrismOptions> agentPrismOptions, ILoggerFactory loggerFactory, AgentPrismMetrics? metrics = null, RunTraceCollector? traceCollector = null, TimeProvider? timeProvider = null, IRunPricingResolver? pricingResolver = null, IRunErrorClassifier? errorClassifier = null, IRunInputStore? runInputStore = null, IServiceProvider? services = null)

Parameters

modelProviders IModelProviderRegistry

optionsMonitor IOptionsMonitor<ModelRunJudgeOptions>

runStore IRunStore

tenantContext ITenantContext

agentPrismOptions IOptions<AgentPrismOptions>

loggerFactory ILoggerFactory

metrics AgentPrismMetrics?

traceCollector RunTraceCollector?

timeProvider TimeProvider?

pricingResolver IRunPricingResolver?

errorClassifier IRunErrorClassifier?

runInputStore IRunInputStore?

services IServiceProvider?

Remarks

The judge’s own call goes through a RunRecordingAgent wrapper and is recorded as RunKind.Eval, like every case run from EvalJobHandler. This computes cost automatically and excludes the row from RunStatistics, so the evaluated agent’s cost does not grow and no new column is needed. The recording decorator deliberately disables quota enforcement with quotaEnforcer: null and event publishing with webhookPublisher: null. A judge is a control-plane cost, does not use the user’s quota, and does not publish its own run.completed noise.

Structured output uses the ChatResponseFormat.ForJsonSchema(JsonElement, ...) overload. Its manually written schema does not use reflection and preserves the AOT stance of AgentPrism.Core.

Properties

Name

The judge’s name. Written into the RunScore.Source field as judge:{Name}.

public string Name { get; }

Property Value

string

Methods

JudgeAsync(RunJudgeContext, CancellationToken)

Scores the run.

public ValueTask<RunJudgment> JudgeAsync(RunJudgeContext context, CancellationToken cancellationToken = default)

Parameters

context RunJudgeContext

The context the judge sees.

cancellationToken CancellationToken

The cancellation token.

Returns

ValueTask<RunJudgment>

The judge’s verdict.