Skip to main content
Evaluators turn a model’s free-text responses into numbers you can compare and aggregate. The library includes rule-based and statistical evaluators, marked Default, and you can create your own LLM-as-a-Judge evaluators to score against criteria you write. You can author an evaluator with nothing else set up, but testing it or running it in an examination needs at least one registered model.

​
Browse the evaluator library

The library is a table, with Name, Description, Type, Created By, Last Updated, Visibility, and Actions. Evaluators SeekrGuard provides show a Default badge under Created By. Above the table, filter by Name, Type, or Model, reorder the list with Sort By, and limit it to your own evaluators with Only mine. Each row’s Actions column offers:
  • View Evaluator – open the evaluator’s detail page.
  • Delete Evaluator – available only for evaluators you created.

​
Create an evaluator

The creation flow builds LLM-as-a-Judge evaluators. The rule-based and statistical evaluators come with SeekrGuard and cannot be created.
An evaluator cannot be changed after you create it.
1

Add Evaluator

On the Evaluators page, click Add Evaluator.
2

Configuration

  • Name – a short label for the evaluator.
  • Description – the judge model relies on this to decide how to evaluate and score outputs. Write it as instructions rather than a label.
  • Evaluator Model – the model that acts as the judge. Optional here, since you pick judge models when you add the evaluator to an examination.
  • Scoring Scale – Minimum, Maximum, Step, and Polarity. The defaults are 0 to 5 in steps of 1, with positive polarity.
  • Constraints (Optional) – natural-language rules for the judge to apply.
Set polarity to match your metric. A metric where high values are bad, such as toxicity, must be negative. Risk scoring inverts non-positive polarity so that higher always means more risk, and a wrong setting makes a model’s risk scores read backwards. See Polarity.
3

Examples (Optional)

Add few-shot examples to calibrate the judge. Each has Text, a Score, and optional Reasoning.
4

Preview & Submit

Review the Name, Model, and the Judge Prompt SeekrGuard generates from your description and constraints, then click Create Evaluator.

​
Read an evaluator’s detail page

View Evaluator opens a page showing the evaluator’s Description, its Inputs, and Facts such as type, dates, score range, and polarity. Inputs lists what the evaluator needs in order to run, such as the model’s output, the expected answer, or retrieved context. Those requirements determine the datasets it can be used with. See Run examinations. The Test Evaluator panel scores text you supply, without creating an examination. Choose a judge model with Select Model, which lists the judge models this evaluator has been used with. Expand Judge Prompt to read the full prompt the judge model receives. Fill in each input the evaluator requires, such as LLM Output, then click Run Test. Load Example fills the fields with sample text and Clear empties them.
Last modified on August 18, 2026