OnlineEvalSummaryService
AgentPrism.Core.dllMaintains a sliding-window summary of online evaluation scores and publishes WebhookEvents.RunScoreLow when its threshold is crossed.
public sealed class OnlineEvalSummaryServiceInheritance
Section titled “Inheritance”object ← OnlineEvalSummaryService
Inherited Members
Section titled “Inherited Members”object.GetType(), object.ToString(), object.Equals(object?), object.Equals(object?, object?), object.ReferenceEquals(object?, object?), object.GetHashCode()
Remarks
Section titled “Remarks”The score window is in memory, with one queue per tenant.
The run_scores table is always the source of truth; an operator can
query its exact result at any time. This service is only an inexpensive live
indicator and alarm, and resets when the process restarts. The same design is
used for RunSampler’s hourly budget (the no-surprises rule: no durable counter store).
One low score does not produce an alarm. Threshold evaluation does not run before OnlineEvaluationOptions.MinSampleSize is met. Models are noisy, and otherwise notifications quickly become ignored.
Constructors
Section titled “Constructors”OnlineEvalSummaryService(IRunStore, IOptionsMonitor<OnlineEvaluationOptions>, IWebhookPublisher?, TimeProvider?)
Section titled “ OnlineEvalSummaryService(IRunStore, IOptionsMonitor<OnlineEvaluationOptions>, IWebhookPublisher?, TimeProvider?)”Maintains a sliding-window summary of online evaluation scores and publishes WebhookEvents.RunScoreLow when its threshold is crossed.
public OnlineEvalSummaryService(IRunStore runStore, IOptionsMonitor<OnlineEvaluationOptions> optionsMonitor, IWebhookPublisher? webhookPublisher = null, TimeProvider? timeProvider = null)Parameters
Section titled “Parameters”runStore IRunStore
optionsMonitor IOptionsMonitor<OnlineEvaluationOptions>
webhookPublisher IWebhookPublisher?
timeProvider TimeProvider?
Remarks
Section titled “Remarks”The score window is in memory, with one queue per tenant.
The run_scores table is always the source of truth; an operator can
query its exact result at any time. This service is only an inexpensive live
indicator and alarm, and resets when the process restarts. The same design is
used for RunSampler’s hourly budget (the no-surprises rule: no durable counter store).
One low score does not produce an alarm. Threshold evaluation does not run before OnlineEvaluationOptions.MinSampleSize is met. Models are noisy, and otherwise notifications quickly become ignored.
Methods
Section titled “Methods”GetSummaryAsync(string, CancellationToken)
Section titled “ GetSummaryAsync(string, CancellationToken)”Returns a tenant’s current window summary and judge cost.
public ValueTask<OnlineEvaluationSummary> GetSummaryAsync(string tenantId, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
cancellationToken CancellationToken
The cancellation token.
Returns
Section titled “Returns”ValueTask<OnlineEvaluationSummary>
RecordScoreAsync(string, int, CancellationToken)
Section titled “ RecordScoreAsync(string, int, CancellationToken)”Adds a newly written judge score to the window and publishes WebhookEvents.RunScoreLow when needed.
public ValueTask RecordScoreAsync(string tenantId, int score, CancellationToken cancellationToken = default)Parameters
Section titled “Parameters”tenantId string
The tenant identifier.
score int
The written score, from 0 to 100.
cancellationToken CancellationToken
The cancellation token.