Search

Flash Cards: The Vocabulary Your Team Needs

Five decks covering the vocabulary a team needs before it can argue productively: the product, classic quality engineering, the AI-driven development lifecycle, evaluating AI, and measurement.

The five flash-card decks and two ways to use them

Five decks, sixty cards. Readable here, printable as cut-out cards.

Every card has a definition and a why it matters line, because a definition you cannot act on is trivia. Click any term below to turn the card over.

Print them
Each deck is also a printable PDF, nine cards to a sheet with the backs mirrored so a double-sided print lines up. Cut along the dashed lines. A stack of cards on a desk gets picked up; a browser tab does not.

How to use these

Alone: one deck a day, five minutes. Read the term, answer before you open the card, and notice which ones you were confident and wrong about — those are the expensive ones.

With a team: read a term aloud and let two people answer before revealing it. Where they disagree, you have found something that belongs in your test strategy rather than in people's heads.

Before a vendor conversation: the Quality Engineering Fundamentals and Evaluating AI decks are the ones that make procurement questions sharper. Several terms that sound standard are not defined by any standards body, and that is worth knowing before you sign something.

Studio Essentials

The product vocabulary. If two people on your team mean different things by "project" or "orchestration", everything downstream of that is confusing.

12 cards. Click a term to reveal the answer.

Test
One scenario described in plain language, which Studio turns into executable steps and then runs against a real browser.
Why it matters: The unit you name, review and own. If you cannot say in one sentence what a test proves, it is not finished.
Project
The container for tests that share an application, environment and settings.
Why it matters: Project boundaries are the first convention to agree when more than one team is involved, because settings are inherited from here.
Orchestration
An ordered group of tests run together, with its own schedule and reporting.
Why it matters: This is where a suite becomes a gate. A test nobody scheduled protects nothing.
Workflow
A scheduled sequence defined by title, steps and schedule, used to run work on a cadence rather than on demand.
Why it matters: Use it for the recurring checks nobody should have to remember to start.
Extension
Custom code you supply — Node, Python, Go or Java — that a test can call when a step needs logic the browser cannot express.
Why it matters: The escape hatch. Reach for it for things like generating a one-time code, not for rewriting a test in code.
Credit
The unit of consumption. Building and running both draw on the same balance.
Why it matters: Consumption is front-loaded: month one looks build-heavy and then flattens. Show the trend, not the snapshot.
Self-heal
Studio re-identifying an element that moved or changed, rather than failing the step.
Why it matters: It covers implementation drift, not behavior change. A heal that quietly binds to the wrong element converts a real defect into a green test — review the healing events, do not just count them.
Timing model
The execution setting that controls how long Studio waits for the application at each step.
Why it matters: The first thing to check when a test is flaky on a slow environment and solid everywhere else.
Execution preset
A named bundle of execution settings, defined on the project and overridable per test.
Why it matters: Presets are how you stop every test carrying its own private timing hacks.
Session
An interactive working context where you build and refine a test with the agent before saving it.
Why it matters: Building and running are different activities with different costs. Know which one you are in.
MCP server
The Model Context Protocol endpoint that lets an external AI tool drive Studio directly.
Why it matters: This is what makes an independent tester agent possible — a second session, on a different model, that only tests.
Report
The execution view: breakdown, trends and pass rate by project.
Why it matters: Pass rate alone is confounded by retries. Read it next to first-run pass rate and flake rate or it will flatter you.

Quality Engineering Fundamentals

Classic vocabulary, stated precisely. These are the terms AI testing vendors blur most often, so knowing the real definitions is directly useful in a procurement conversation.

12 cards. Click a term to reveal the answer.

Test oracle
The source that determines the expected result. Not the assertion — the assertion is only the comparison that consults it.
Why it matters: An "AI that decides pass or fail" is a pseudo-oracle at best. If your oracle is another model, you have no independent source of truth.
Test oracle problem
The difficulty of determining whether a test passed or failed for a given input and state.
Why it matters: The central challenge in testing AI, and the reason metamorphic testing exists.
Flaky test
A test whose outcome is non-deterministic with respect to a given software version — it passes and fails with no change to the code.
Why it matters: Neither ISTQB nor ISO defines this term, which is exactly why "flaky" becomes the bin where real race conditions get filed and forgotten.
Regression testing
Change-related testing that looks for defects introduced or uncovered in the UNCHANGED areas of the software.
Why it matters: Defined by scope and trigger, not by size or schedule. Your nightly "regression suite" may contain very little regression testing.
Confirmation testing
Testing after a fix, to confirm the failure does not recur. ISO calls it retesting.
Why it matters: Confirmation targets the changed thing; regression targets the unchanged things. Most teams erase the distinction by filing both in one suite.
Coverage item
The attribute a coverage percentage is measured against.
Why it matters: "We have 80% coverage" is a malformed claim under both ISTQB and ISO until you say 80% of what.
Escaped defect
A defect not caught by a test activity that was supposed to find it.
Why it matters: The definition hinges on intent, so escape counts only mean something against an explicit statement of what you meant to cover.
Critical journey coverage
The share of business-critical end-to-end journeys that have a reliable automated check.
Why it matters: The one coverage number an executive can actually interpret, because it maps to business risk rather than to code structure.
Shift left
Performing testing and quality activities as early as possible in the lifecycle.
Why it matters: It is a cost trade, not a free win: more effort earlier, savings later. It also means more than "run tests in CI".
Shift right
Testing a system continuously in production.
Why it matters: Not optional for anything containing a model, because drift is only observable against real traffic.
Risk-based testing
Selecting and prioritizing test effort against analyzed risk types and levels.
Why it matters: The ISO wording presupposes a documented risk analysis. Gut-feel prioritization does not qualify, however confidently it is asserted.
Metamorphic testing
Asserting a relationship between outputs instead of a single expected output — for example, that translating a longer sentence should not produce a shorter result.
Why it matters: The main principled answer to the oracle problem when you cannot state the one correct answer.

The AI-Driven Development Lifecycle

The vocabulary of building software with agents, and where verification sits inside it.

12 cards. Click a term to reveal the answer.

AI-DLC
The AI-driven development lifecycle: an agent drives the work across the lifecycle while people validate and decide.
Why it matters: A vendor coinage from 2025, not a standard. Its own sources describe either three phases or five. Adopt it deliberately rather than comply with it.
Verification gate
A check an agent cannot mark its own work past.
Why it matters: Without one, "looks done" is the only stopping signal available, and you become the verification loop.
Verification tax
The effort that moves from writing code to checking it once an agent is generating the code. DORA's own term.
Why it matters: It names where the savings went, which makes it the most useful single concept for justifying an independent test layer.
Delivery stability
How often a change causes a failure, and how long recovery takes.
Why it matters: The measure where AI adoption shows a NEGATIVE relationship in DORA's data. If you track only throughput, you will not see the cost.
Spec artifacts
The named files carrying requirements into agent execution — typically a requirements file, a design file and a task list.
Why it matters: Your new traceability artifacts, and they are version controlled, so coverage can trace to a file rather than a ticket.
Agentic coding
An agent planning and executing multi-step code changes rather than completing a line at a time.
Why it matters: Volume of change rises faster than review capacity. That gap is the whole problem.
The independent tester principle
Never let the session that built the thing be the session that verifies it.
Why it matters: An agent that runs your suite treats green as its stopping condition, so weak tests stop merely failing to catch bad code and start licensing it.
Adversarial review
A second agent, ideally on a different model, whose job is to find what the first one missed.
Why it matters: Different training, different failure modes. A reviewer sharing the builder's blind spots is not a review.
Context window
Everything the model can reference at once — system prompt, conversation, tool results and its own output.
Why it matters: A long-running agent's reliability decays over its own run. Test at realistic context occupancy, not on short synthetic cases.
Perceived versus actual productivity
The gap between how much faster people feel and how much faster they are.
Why it matters: In a randomized trial, developers took 19% longer with AI tools while believing they had been 24% faster. Pair every survey number with a measured one.
Model Context Protocol (MCP)
An open protocol connecting AI applications to external tools and data.
Why it matters: How an external agent drives Studio. Note the current revision is stateless, and sampling, roots and logging are deprecated.
Guardrail
A constraint enforced outside the model, so it holds regardless of what the model decides.
Why it matters: A prompt is a request. A guardrail is a control. Only one of them is testable.

Evaluating and Testing AI

For when the thing under test is itself a model — where a test case is a distribution rather than a pass or a fail.

12 cards. Click a term to reveal the answer.

Eval
A structured test of model output against defined criteria, run over a curated set of cases.
Why it matters: The AI equivalent of a test suite, except the unit of judgment is a rate across runs, not a single outcome.
LLM-as-a-judge
Using a strong model to score another model's output against a rubric or reference.
Why it matters: On the benchmark that named the technique, the judge agreed with expert humans 85% of the time against 81% human-to-human. Useful, and itself a component under test.
Position bias
A judge preferring whichever candidate appears first, or second, regardless of quality.
Why it matters: Fixed by test design: run both orderings and require a consistent verdict.
Non-determinism
The same input producing different output on repeated calls.
Why it matters: Temperature zero does NOT buy a reproducible test — production endpoints vary with server-side batch size. "It passed once" is not evidence.
pass@k
The probability that at least one of k attempts succeeds.
Why it matters: Optimistic by construction. Useful for code generation, misleading as a reliability claim.
pass^k
The probability that ALL k attempts succeed.
Why it matters: The one that matters for anything an agent does unattended. The two are read aloud identically, so write them down.
Groundedness
Whether an answer is supported by the retrieved context it was given.
Why it matters: Distinct from factuality. An answer can be true and ungrounded, which is still a failure of the retrieval system.
Hallucination
Confident output not supported by the source or by fact.
Why it matters: Not a bug you fix once. Treat it as a measured rate with a threshold, like any other quality property.
Red teaming
Structured adversarial probing for harmful or policy-violating behavior.
Why it matters: Exploratory testing for harm. Its findings are the raw material for your safety regression suite — it is not itself a coverage claim.
Prompt injection
Input that alters behavior because instructions and data share one channel.
Why it matters: The highest-severity class for any tool-using agent. Indirect injection — from a page, ticket or document the system reads — is the harder half.
Drift
Change over time in input distribution (data drift) or in the relationship between input and outcome (concept drift).
Why it matters: The failure mode with no equivalent in classic software: a system that passed every test at release degrades with nobody changing a line.
Golden dataset
A hand-curated, expert-labeled set whose expected outputs are trusted as correct.
Why it matters: No standards body and no major vendor defines the phrase. The citable term is ground truth, and its label quality caps every metric you compute from it.

Measuring and Proving Value

The five numbers, and the traps in the ones people reach for instead.

12 cards. Click a term to reveal the answer.

Escaped defects
Defects that reached production despite a test activity meant to catch them.
Why it matters: The outcome measure. Everything else on this card deck is a leading indicator of this one.
Time to feedback
How long after a change lands a developer learns whether it broke something.
Why it matters: A comprehensive but slow suite gets bypassed. This often predicts real quality better than coverage does.
Maintenance share
The proportion of automation effort spent repairing existing tests rather than extending coverage.
Why it matters: The number that decides whether a program compounds or stalls — and the claim vendors make most often, with no standard definition behind it.
Flake rate
The share of results that are non-deterministic for an unchanged version.
Why it matters: Above roughly 2%, people stop reading red. At that point the suite has stopped working regardless of what it covers.
Baseline
What was true before you changed anything, written down, with the date on it.
Why it matters: Half an hour in week one. Without it you will improve things and be unable to prove it — and this is the step teams skip.
DORA metrics
Change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate.
Why it matters: DORA has retired MTTR, which expanded three different ways in three different standards. Correct this in any metrics deck you inherit.
Test count
How many tests you have.
Why it matters: Not a result. It is the easiest number to move without improving anything, and agents make it trivially easy. Report what is proven, not what was written.
Pass rate
The share of executed tests that passed.
Why it matters: Confounded by the thing it is used to hide. With automatic retries it measures your retry policy as much as your product.
J-curve of value realization
The dip before the gain: learning curve, verification tax, and adjusting downstream process to higher code volume.
Why it matters: Useful for setting expectations with a sponsor. The dip is the tuition, and it is predictable rather than a sign of failure.
Hours displaced
Manual test execution hours no longer being spent.
Why it matters: The number a finance conversation actually turns on. Capture it per release, not per month, so it survives a schedule change.
Critical journey
A path a customer takes to do the thing you charge them for.
Why it matters: Usually ten to twenty. If your list has sixty, you have listed features. Test it: if this broke at 2am Friday and nobody noticed until Monday, what would it cost?
Coverage of what matters
The share of critical journeys with a reliable automated check.
Why it matters: The question to ask of any suite: if every test passes, what do we now know is true?

Where these come from

The definitions are condensed from AI Testing and AI-DLC: A Glossary, which carries the full versions with primary sources and flags the terms the industry genuinely disagrees about. When a card feels too short, that is where the rest of it is.

Ready to check yourself? Check Yourself: Twenty Questions on Testing With AI.