Evaluating and Testing AI: Terms

A/B testing of models

also: champion/challenger, side-by-side

Splitting live traffic between two model or prompt configurations and comparing outcome metrics — task success, thumbs-up rate, conversion, escalation rate.

Why it matters for testing: Offline scores and human preference frequently disagree, so this is the tiebreaker before a full rollout. It needs pre-declared metrics and sample sizes like any experiment.

Adversarial prompting

also: adversarial suffix, automated attack

Inputs deliberately crafted to break behavior, including automatically optimized ones that transfer across models.

Why it matters for testing: Because attacks are automated and transferable, manual adversarial cases go stale. Treat attack generation as a tool in your harness, not a one-off exercise.

Answer relevance

also: relevance, answer relevancy

Whether the response actually addresses the question asked, independent of whether it is true. Low scores flag off-topic, evasive, vague or partial answers.

Why it matters for testing: Catches the common failure where the model returns something accurate but useless — invisible to correctness-only assertions.

Attribution (AIS)

also: Attributable to Identified Sources

A formal framework for whether every claim about the external world in an output is verifiable in the identified source, with a two-stage human annotation protocol.

Why it matters for testing: Gives you a defensible, auditable definition when "is this grounded?" has to survive a compliance or legal review.

Benchmark

A standardized, published eval set plus scoring protocol, used to compare models — MMLU, GSM8K, SWE-bench, IFEval.

Why it matters for testing: Benchmarks tell you about a model; they almost never tell you about your application. Use them to choose a model, never as a substitute for product evals.

Calibration

Whether stated or implied confidence matches observed accuracy — a model claiming 80% confidence should be right about 80% of the time.

Why it matters for testing: Two uses: calibration of the system enables abstention and escalation thresholds; calibration of a judge determines whether its scores mean anything.

Chunk attribution and utilization

Attribution is binary per chunk: did this retrieved chunk actually affect the response? Utilization is continuous: how much of that chunk's text mattered.

Why it matters for testing: Low values mean you are paying tokens and latency for context the model ignored — a measurable tuning target for chunk size and top-k.

Usage is contested. These are product-specific terms rather than an industry standard.

Citation accuracy

also: citation precision and recall, verifiability

Whether inline citations actually support the sentences they are attached to. Split into citation recall (the cited passages fully support the sentence) and citation precision (each individual citation supports it).

Why it matters for testing: Citations are the user's trust surface, and a plausible-but-unsupporting citation is worse than none. Assert on citation support, not citation presence.

Coherence

also: fluency

Whether the response is well-organized, internally consistent and readable end to end. Offered as a standard judged metric by evaluation services.

Why it matters for testing: Near-perfect scores are normal for frontier models, so a coherence drop is a strong smoke signal of a broken prompt, truncation, or a decoding bug.

Confabulation

In the research literature, a subset of hallucinations: arbitrary, incorrect generations that are sensitive to random seed, so the model produces different wrong answers to the same question.

Why it matters for testing: The narrow sense is directly testable — high answer variance across repeated runs of the same input is a detectable signal of likely fabrication.

Usage is contested. Outside that literature, many practitioners use it as a straight synonym for hallucination, sometimes preferring it as less anthropomorphic.

Context relevance

also: context precision

Whether the retrieved chunks are actually useful for the question, judged independently of what the model then did with them.

Why it matters for testing: Localizes the defect to the retriever, index or chunking layer — so you fix embeddings or chunk size instead of re-prompting the model.

Data contamination

also: test-set leakage, benchmark contamination

Evaluation data, or near-duplicates of it, appearing in the model's training corpus — inflating scores without real capability.

Why it matters for testing: A public benchmark your vendor may have trained on is not a valid acceptance test. Held-out, privately authored cases are.

Eval

also: evaluation, LLM test

A test for an AI system: supply an input, then apply grading logic to the output. The vocabulary is deliberately QA-like — a task is one test case, a trial is one attempt at it, a trace is the full record, and an outcome is the end state.

Why it matters for testing: It is the LLM world's word for "test", with one crucial difference: a single task usually needs several trials, so a test case is a distribution rather than a pass/fail row.

Eval set

also: eval dataset, test set, task suite

The curated collection of tasks an eval runs over, each with inputs plus expected outputs or grading criteria. Good practice mixes benchmark-style cases with samples of real production traffic.

Why it matters for testing: Its composition is your coverage story. A suite drawn only from synthetic happy paths will pass while production fails.

Evaluation harness

also: eval framework

The infrastructure that runs evals end to end: loads tasks, executes trials, records every step, applies graders and aggregates results.

Why it matters for testing: It is your test runner, and the same requirements apply — reproducible configuration, recorded artifacts, results comparable across versions.

Factuality

also: factual accuracy

Correctness against the world, not merely against the provided context — typically measured with short fact-seeking questions graded against reference answers.

Why it matters for testing: A response can be perfectly faithful to a wrong retrieved document, so factuality and groundedness have to be tested separately.

Faithfulness

The same idea as groundedness, expressed as a ratio: the fraction of claims in the answer that are supported by the retrieved context.

Why it matters for testing: It isolates generation failures from retrieval failures, which is the first fork in any retrieval bug triage.

Usage is contested. Groundedness, faithfulness, context adherence and attribution are near-synonyms across vendors with differing formulas. Pin the framework, not the word.

Flakiness in LLM outputs

A test that passes on some trials and fails on others with no change. In traditional QA a flaky test is a defect in the test; with LLMs, output variance is an inherent property of the system under test, so a task legitimately has a pass rate.

Why it matters for testing: Do not quarantine or retry until green. Convert single-trial assertions into n-trial rates with a declared threshold, or the suite becomes untrustworthy.

Golden dataset

also: golden set, gold standard set

A hand-curated, expert-labeled eval set whose expected outputs are trusted as correct and used as the reference for everything else.

Why it matters for testing: It is the equivalent of a trusted regression baseline. Its label quality caps how far you can trust every metric computed from it.

Usage is contested. No standards-body definition exists. Teams disagree on whether a golden set must carry exact expected outputs or only grading criteria.

Grader

also: scorer, evaluator

The logic that scores one aspect of a trial. Three families: code-based (string match, schema checks, tool-call verification), model-based (rubric, pairwise, reference-based) and human.

Why it matters for testing: Prefer a code-based grader whenever the property is mechanically checkable — faster, free and deterministic. Reserve judges for what genuinely needs judgment.

Ground truth

also: reference, gold label

The authoritative correct answer for an input, established by humans, by a deterministic system, or by a verifiable check. For generative tasks it is often a set of acceptable answers, or a rubric, rather than one string.

Why it matters for testing: Exact-match assertions against a single ground-truth string cause false failures in generative testing. You usually need semantic or rubric comparison instead.

Groundedness

Whether every claim in the output is attributable to the supplied source context. Usually implemented as a judged metric returning a score plus an explanation of which claims were unsupported.

Why it matters for testing: For any retrieval or document-Q&A feature this is the primary correctness assertion — and unlike "is it true?", it is checkable against a bounded input.

Guardrails

also: rails

Runtime controls wrapped around the model rather than trained into it — input, dialog, retrieval, execution and output rails that can call classifiers, redact data or block a flow.

Why it matters for testing: Guardrails are testable software with their own false-positive and false-negative rates. Test the rails and the model separately, then together.

Hallucination

Output that is fluent and confident but not supported by the source material or by fact. A 2025 result frames it as a consequence of statistical training plus the fact that most evals score guessing higher than abstaining.

Why it matters for testing: A defect class with no stack trace. You can only find it by grading content against a reference or a source — and your own scoring rules may be rewarding it.

Human preference data

also: preference pairs, comparison data

Datasets of prompt, response A, response B and which is better, collected from humans. Used to train reward models and to validate automated judges.

Why it matters for testing: It is the root ground truth for "quality", so its known biases — length, agreeableness, formatting — propagate into models and into judges.

Instruction following

Whether the model obeys explicit constraints in the prompt — format, length, inclusions, exclusions. Made objectively gradable by verifiable instructions such as "at least 400 words".

Why it matters for testing: Verifiable instructions are the one part of LLM behavior you can assert on with plain deterministic code. Cheap, stable, no judge required.

Inter-rater reliability

also: IRR, inter-annotator agreement

The degree to which independent graders assign the same labels, corrected for chance agreement.

Why it matters for testing: If human raters cannot agree on a criterion, no judge can be validated against it and no eval built on it means anything. Measure this before trusting labels.

Jailbreak

A prompt that gets a model to bypass its own safety training and produce content it would normally refuse.

Why it matters for testing: Jailbreak suites are perishable — new techniques appear continuously, so this is continuous testing, not a gate you pass once.

Usage is contested. Routinely conflated with prompt injection, including in vendor material. The distinction: jailbreaking subverts the model's policy, injection subverts the application's instructions.

LLM-as-a-judge

also: model-based grader, AI judge

Using a strong model to score another model's output against a prompt, rubric or reference. On the benchmark that named the technique, a GPT-4 judge agreed with expert human preference 85% of the time, against 81% for human-versus-human agreement on the same votes.

Why it matters for testing: The only scalable way to grade open-ended output, and itself a component under test: judges need their own validation set and version pinning.

Non-determinism

The same input can produce different outputs on repeated calls. This is not only sampling: production endpoints are non-deterministic even at temperature 0, because server-side batch size varies with load and the underlying kernels are not batch-invariant.

Why it matters for testing: Setting temperature 0 does not buy you a reproducible test. Plan for repeated trials, and know that "it passed once" is not evidence. Measured effect: across five models and eight tasks, accuracy varied by up to 15% between runs of an identical configuration, and no model returned identical output strings.

Usage is contested. The batch-invariance explanation comes from a lab engineering blog post rather than peer-reviewed work — credible and widely cited, but not refereed. The peer-reviewed citation for non-determinism in evaluation is Song et al., NAACL 2025.

Offline evaluation

also: pre-deployment eval

Running evals against a fixed dataset before release, in development or CI, to gate a change. The direct analogue of a regression suite in the pipeline.

Why it matters for testing: Offline results license exactly one claim: "good on this test set". Necessary, never sufficient.

Online evaluation

also: continuous evaluation, production eval

Scoring real production interactions continuously, usually on a sampled fraction of traffic, with the same graders used offline.

Why it matters for testing: The only way to catch quality drift caused by things absent from your test set — changed user behavior, changed upstream data, a silently updated model.

Pairwise comparison

also: side-by-side, head-to-head

Showing a grader two candidate responses to the same prompt and asking which is better, rather than asking for an absolute score. Aggregated into per-model ratings.

Why it matters for testing: Relative judgments are far more reliable than absolute 1–5 scores, so this is the method for "is the new version better?" — but it gives no absolute bar.

pass@k

also: pass at k

The probability that at least one of k sampled attempts is correct, estimated from n ≥ k samples per task rather than by naively sampling k.

Why it matters for testing: It measures "can it get there with retries", which flatters a system used once. Report pass@1 too if your product has no retry loop.

pass^k

also: pass hat k, all-of-k

The probability that all k trials succeed — the strict counterpart to pass@k.

Why it matters for testing: This is the metric that matches an unattended production path, where one bad trial is a customer-visible failure.

Position bias

A judge preferring whichever candidate appears first, or second, in a pairwise prompt regardless of quality. Documented alongside verbosity bias.

Why it matters for testing: The fix is a test-design fix: run both orderings and require a consistent verdict, or randomize order. Otherwise your judge is encoding an artifact.

Prompt injection

An input that alters the model's behavior because instructions and data share one channel. Direct injection comes from the user's own input; indirect injection comes from content the system ingests — a web page, document, ticket or email carrying hidden instructions.

Why it matters for testing: The highest-severity class for any tool-using or retrieval-based agent. Testing it means planting payloads in data sources, not just in the chat box.

Red teaming

Structured adversarial probing to discover, measure and reduce harmful outputs, by humans or by other models.

Why it matters for testing: Exploratory testing for harm. Its findings are the source material that becomes your safety regression suite.

Refusal rate

also: over-refusal, false refusal, exaggerated safety

How often the model declines to answer. The testable pair is over-refusal rate (refusals on safe prompts that merely look unsafe) against safety rate (refusals on genuinely unsafe prompts).

Why it matters for testing: Safety tuning and usefulness trade off directly, so a release that improves safety numbers must be regression-tested for new false refusals.

Regression eval

also: regression testing for models

A suite of tasks the system already handles, expected to sit at or near a 100% pass rate, run on every prompt, model or tool change. Contrast capability evals, which probe hard behaviors and should start low.

Why it matters for testing: The split matters operationally: a capability eval at 40% is healthy, a regression eval at 97% is an incident.

Retrieval precision and recall

Classic information-retrieval measures applied to the retriever: precision is the fraction of retrieved chunks that are relevant, recall the fraction of relevant chunks that were retrieved.

Why it matters for testing: Recall caps answerability — missing evidence guarantees a wrong answer. But check which definition your tool implements: some frameworks compute judged, rank-aware variants rather than the textbook formulas.

Rubric grading

also: criteria-based scoring

Scoring against written criteria instead of a reference answer, either a fixed rubric or criteria generated per prompt as a set of pass/fail checks.

Why it matters for testing: Vague rubrics are the number-one cause of unstable judge scores. Write them as discrete binary assertions and version them like test code.

Safety filter

also: content classifier, moderation model

A classifier applied to the prompt and/or the response against a risk taxonomy.

Why it matters for testing: It is a binary classifier, so evaluate it like one: confusion matrix, threshold choice, and the cost asymmetry between a miss and a false block.

Sample size for evals

also: statistical power, minimum detectable effect

How many eval questions, and trials per question, you need to reliably detect a difference of a given size.

Why it matters for testing: Most "the new prompt is 2% better" claims sit inside the noise band of a 100-question eval. Compute the minimum detectable effect before you gate on it.

Scenario testing

also: behavioral testing

Testing realistic multi-turn situations rather than isolated prompts. In benchmark design the word also means a formally specified use case — task, domain, language, user — measured on several metrics.

Why it matters for testing: Scenario-level tests catch failures that per-prompt assertions structurally cannot: state carried across turns, recovery after an error, tool-call sequencing.

Usage is contested. Two established and quite different uses; check which one a colleague means.

Self-preference bias

also: self-enhancement bias

An AI judge rating its own generations more highly than humans rate them. Models can recognize their own text, and that ability correlates with the strength of the bias.

Why it matters for testing: Never let a model be the sole judge of its own family's output in an acceptance gate. Use a different judge model, or a human sample.

Usage is contested. Later work argues part of the effect reflects genuine quality differences rather than pure bias. Note also that the paper which named the bias said in the same breath that its data could not determine whether the effect was real; the evidence for it comes from the later study cited here.

Sycophancy

The tendency to tell the user what they appear to want — agreeing with a stated belief, caving to pushback, or revising a correct answer when challenged. Traced to preference data in which human raters rewarded agreement.

Why it matters for testing: Single-turn evals miss it entirely. You need multi-turn tests that assert the model holds a correct answer under disagreement.

Temperature

A decoding parameter controlling randomness in token selection. Near zero yields the most consistent output; higher values more varied output.

Why it matters for testing: Match it to intent — low for extraction, classification and regression tests — and always record it with the result, since it is part of the test configuration.

Toxicity

Rude, disrespectful or unreasonable content likely to make someone leave a conversation. Measured with a prompt set plus a classifier score.

Why it matters for testing: Results depend on the classifier version and the chosen cut-off. Record both, or your numbers will not reproduce.

Variance across runs

also: run-to-run variance, standard error

The spread in scores across repeated trials and across eval questions. Best practice is to treat eval questions as a sample, report standard errors, and compare models on paired per-question differences.

Why it matters for testing: Paired, question-level comparison detects far smaller real differences than comparing two aggregate scores — the same reason you compare before and after on identical fixtures.

Verifier pattern

also: generate-and-verify, best-of-n reranking

Generate several candidate outputs, then use a separate verifier — a model, a test suite, or a code check — to select or reject.

Why it matters for testing: The architectural version of testing. When a property is cheaply checkable, a verifier converts an unreliable generator into a reliable feature.