A
aggregation method – how a risk category combines its evaluators’ scores into a single category score, such as a weighted average. Distinct from the method a risk profile uses to combine its categories. See Build risk profiles. Assistant Message – the message template that holds an expected output. ↑ Back to topB
BLEU – a metric that scores how much of a model’s output matches the expected output, counting runs of consecutive words. Higher scores mean the output stays closer to the reference wording. See Create evaluators. ↑ Back to topC
Chat History – the message template that holds conversation history. comparison – a scoring of a single response by several evaluators and judge models, shown as a grid. By contrast, an examination runs over a dataset. See Run comparisons. Context – the message template that holds a context input. context input – background or retrieved material supplied alongside the user’s question, used in retrieval-augmented generation (RAG). When you upload a dataset, you map it to the Context template. See Add and manage datasets. conversation history – prior turns of a multi-turn conversation included with an input. When you upload a dataset, you map it to the Chat History template. See Add and manage datasets. ↑ Back to topD
dataset – a collection of test inputs, and optionally expected answers, that you test models on. Each input can include a system prompt, user question, context, conversation history, and expected output. See Add and manage datasets. ↑ Back to topE
evaluator – a scorer that grades a model’s output on one dimension, such as accuracy, toxicity, or relevance. There are three types: LLM-as-a-Judge, rule-based, and statistical. See Create evaluators. examination – a batch run that scores one or more models on a dataset using one or more evaluators, then reports aggregated statistics, per-row scores, and charts. A run is queued, then in progress, then completed or failed. See Run examinations. expected output – the reference or ground-truth answer for an input, which evaluators compare the model’s output against, using methods such as exact match, BLEU, or ROUGE. When you upload a dataset, you map it to the Assistant Message template. See Add and manage datasets. ↑ Back to topJ
judge model – a model that scores a response rather than producing one. An LLM-as-a-Judge evaluator supplies the criteria and a judge model applies them. In an examination you choose judge models for each evaluator; in a comparison you choose them in Judge Models. See Run comparisons. ↑ Back to topL
LLM-as-a-Judge – an evaluator you create that uses one or more other LLMs to score a model’s output against your criteria. You pick those models when you add it to an examination. See Create evaluators. ↑ Back to topM
max tokens – the maximum length, in tokens, of a model’s response. On the Chat page, you can set it between 128 and 4096. See Chat with a model. memory – on the Chat page, the Use conversation history toggle. When on, the model sees all previous messages in the conversation. When off, each message is independent. See Chat with a model. method (risk profile) – how a risk profile combines its category scores into an overall risk score: weighted linear, with penalties, or multiplicative. See Build risk profiles. model – a model you register in SeekrGuard from a provider, such as GPT-4 or Llama. Its type is foundational or fine-tuned, and its status is active, queued, or inactive. See Register and manage models. model card – a model’s profile page showing its details, a risk scorecard for a risk profile you select, and analysis tabs. See Register and manage models. ↑ Back to topP
polarity – a property of an evaluator’s scoring scale that indicates which direction is better. Positive means higher scores are better outcomes; negative means higher scores are worse. When scores are combined into risk, negative-polarity ones are flipped so that higher always means more risk. See Create evaluators. provider – the source of a model. When you register models you choose OpenAI or Seekr, which determines the models available to add. In the model list, the Provider column shows the organization each model came from, such as Google or Seekr. See Register and manage models. ↑ Back to topR
risk category – a named risk dimension, such as factual accuracy, built by pairing one or more evaluators with the datasets they run against. Its aggregation method combines those scores into one category score, which a risk profile then weights. See Build risk profiles. risk level – how a model’s risk score is rated, from least to most risk: low, medium, high, or critical. In the Model Risk Assessment section of the Home page, it appears as a badge on each model’s card and as a filter above them. See Build risk profiles. risk profile – a weighted set of risk categories for a particular use case, which produces an overall risk score and risk level for each model. A profile can be marked active, and the Model Risk Assessment section on the Home page scores models against an active profile you select. See Build risk profiles. risk score – a model’s overall risk under a risk profile, on a 0–100 scale where higher means more risk. It maps to a risk level and appears on each model’s card in the Model Risk Assessment section of the Home page. See Build risk profiles. ROUGE – a metric that scores how much of the expected output a model’s response covers. Higher scores mean more of the reference wording appears in the output. See Create evaluators. rule-based evaluator – an evaluator that applies fixed rules, such as pattern or constraint checks. Provided by SeekrGuard, with a Default badge under Created By in the evaluator library. See Create evaluators. ↑ Back to topS
sample selection – the portion of a dataset an examination processes. When you create an examination, set it in the Sample Selection control as a percentage of the dataset or a number of samples; the control shows how many of the total samples will be used. See Run examinations. scoring scale – the range of scores an evaluator can produce, set by its Minimum, Maximum, and Step, along with its polarity. When you create an LLM-as-a-Judge evaluator, you define it in the configuration step. See Create evaluators. statistical evaluator – an evaluator that computes a metric, such as ROUGE, BLEU, or exact match, usually against an expected output. Provided by SeekrGuard, with a Default badge under Created By in the evaluator library. See Create evaluators. System Message – the message template that holds a system prompt. system prompt – an instruction that guides a model’s behavior, such as You are a helpful assistant. When you upload a dataset, you map it to the System Message template. On the Chat page, System Prompt is one of the generation settings. See Add and manage datasets and Chat with a model. ↑ Back to topT
temperature – a generation setting that controls randomness in a model’s responses: lower is more deterministic, higher is more varied. On the Chat page, you can set it between 0 and 1. See Chat with a model. ↑ Back to topU
user input – the user’s question or query in a dataset input. When you upload a dataset, you map it to the User Message template. An evaluator that requires a user input can only run against datasets that supply one. See Add and manage datasets. User Message – the message template that holds a user input. ↑ Back to topV
visibility – who can see and edit a resource:- Private – only you.
- Public (view only) – everyone can see it, but no one else can edit it.
- Public (collaborative) – everyone can see it, and others can edit it.