Provide inputs
An evaluation draws on three inputs:- Model – the model under test, such as GPT-4 or Llama, registered from a provider. See Register and manage models.
- Dataset – the inputs you run the model on, optionally with expected answers. See Add and manage datasets.
- Evaluator – a scorer that grades each output on one dimension, such as accuracy, toxicity, or relevance. See Create evaluators.
Run evaluations
You put the inputs to work in two ways:- Examination – a batch run that scores one or more models across all or part of a dataset and reports aggregated results. See Run examinations.
- Comparison – a focused check that scores a single response with several evaluators and judge models at once. See Run comparisons.
- Chat – talk to a model directly and score any response on the spot.
Assess risks
Individual evaluator scores become useful once they combine into a single, weighted measure of risk:- Risk category – a named dimension, such as factual accuracy, backed by one or more evaluators paired with the datasets they run against. See Create risk categories.
- Risk profile – a weighted combination of risk categories for a specific use case. See Build risk profiles.
- Risk score and risk level – scoring a profile against a model yields a score from 0 to 100, where higher means more risk, rated from low to critical.