Reference

AI evals and LLM observability glossary

18 terms buyers meet when evaluating AI eval tools.

Eval (evaluation): A repeatable test that scores AI outputs against defined criteria.
Offline eval: An evaluation run on a fixed dataset before a change ships, often in CI.
Online eval: An evaluation that scores live production traffic or traces as they arrive.
LLM-as-a-judge: Using a language model, prompted with grading criteria, to score another model’s output.
Scorer (evaluator): A function, rule, or judge prompt that assigns a score or label to an output.
Trace: The recorded path of one request through an AI application, including model calls and tool calls.
Span: One step inside a trace, such as a single LLM call or tool call. Many tools price by spans.
Eval dataset: A set of inputs, and often expected outputs, used to run offline evals.
Experiment: One run of an application version over a dataset, saved so scores can be compared across versions.
Guardrail: A real-time check that blocks, flags, or changes an input or output before it reaches the user.
Red teaming: Adversarial testing to find unsafe, insecure, or unwanted behavior, such as jailbreaks or prompt injection.
Prompt injection: An attack that hides instructions in input data to make a model ignore its intended instructions.
OpenTelemetry (OTel): An open standard for collecting traces and metrics. Many eval tools accept OTel data.
BYOC (bring your own cloud): A deployment where the vendor runs the control plane and your cloud account stores the data.
Source-available license: A license that publishes source code but restricts some uses, unlike OSI open-source licenses. Elastic License 2.0 is one example.
SOC 2 Type II: An independent auditor’s report on how a company’s security controls operated over a period.
ISO/IEC 27001: An international certification for an information security management system.
BAA (business associate agreement): A HIPAA contract that lets a vendor handle protected health information for a covered entity.