Judge rubrics · New

New judge rubric.

A rubric carries a prompt template + dimensions + scale + LLM provider. The eval runner invokes it per-case after the case lands and persists one row per dimension.

Agent (optional)Leave as global to allow any agent in the org to opt into this rubric. Pin to an agent to restrict eligibility.NameDescription (optional)Prompt templateSubstitutions available: {{conversation}}, {{expected}}, {{actual}}, {{dimensions}}. The judge must return strict JSON{scores: { dim: number }, reasoning: { dim: string }}; parse failures persist a sentinel score and skip case grading without affecting pass / fail.

Dimensions

groundednesscitation_precisiontonerefusal_correctness

Each dimension is scored by the LLM judge. 1..16 entries; 1-64 chars each.

ScaleLLM provider config idNo LLM provider config exists yet. Configure one first via PUT /v1/providers/llm.Model override (optional)Falls back to the provider's default model when blank.

Cancel