The vocabulary of AI testing is moving faster than anyone's ability to standardize it. Terms get coined in a blog post, picked up by vendors, and are load-bearing in a procurement document six months later — often meaning three different things to the three people in the room.
This glossary is our attempt to fix that for our own market. Every term carries a definition, a note on why it matters if you are responsible for software quality, and a link to a primary source. Where the industry genuinely disagrees, we say so rather than picking a side.
IMG_01
The four pages
| The AI-Driven Development Lifecycle | Lifecycle framings, the practices of building with agents, the artifacts they read and write, and where verification gates actually sit. |
| Agentic Systems | How agents are built, what they remember, how they connect to your systems through MCP, and the named ways they fail. |
| Evaluating and Testing AI | Evals, judges, groundedness and hallucination, red teaming — and the statistics you need when a test case is a distribution rather than a pass or a fail. |
| Quality Engineering and AI Assurance | Classic QE vocabulary restated precisely, plus the governance and regulatory terms that now touch testing. |
About AI-DLC, honestly
You will increasingly hear AI-DLC — the AI-driven development lifecycle — used as the frame for how software gets built now. It is a useful frame, and it is worth being precise about what it is.
- It is a vendor coinage, not a standard. It originates with AWS in July 2025. No standards body has adopted it. The only standardized lifecycle baseline remains ISO/IEC/IEEE 12207.
- Its phase count is ambiguous. The methodology describes three phases — Inception, Construction, Operations. The open-source implementation ships five phases and 33 stages. "The AI-DLC phases" means nothing without saying which.
- A colliding acronym exists. Elsewhere, "AI development lifecycle" means the lifecycle of building AI systems — the MLOps-adjacent sense. A strategy written against the wrong one aims at model drift instead of generated-code defects.
- Analysts use a different word. Gartner's term for the same shift is AI-native software engineering. That is the label likely to appear in a board deck, where AI-DLC will not.
None of that makes the idea less useful. It makes it a frame to adopt deliberately rather than a standard to comply with.
Why the vocabulary matters more than it looks
56 of the 191 terms here are flagged as contested — genuinely disputed, framework-relative, or used by vendors for whichever meaning suits the product. That proportion is the finding, not a caveat.
A few that cost real money when they are misread:
- Replay does not mean reproduction. In every current agent framework it means re-execution from a checkpoint — steps after the checkpoint re-run and may produce different results. Deterministic replay of an agent run is not achievable with today's tooling, so any test strategy premised on it needs rethinking.
- Temperature zero does not give you a reproducible test. Production endpoints are non-deterministic even at zero, because server-side batch size varies with load. "It passed once" is not evidence.
- Tool annotations are hints, not guarantees. In MCP, the idempotency hint defaults to false and the specification says annotations from untrusted servers must be treated as untrusted. Agents retry, so a non-idempotent tool will eventually be called twice.
- The EU AI Act's high-risk deadline has moved. It was postponed in July 2026 to December 2027 and August 2028. A great deal of published material still cites August 2026.
- ISTQB does not define "flaky", "self-healing", "autonomous testing", "code coverage", "fairness" or "model drift" — every one of them a term AI testing vendors lean on heavily. That gap is worth knowing before you write one into a contract.
What the evidence says about quality in an AI-DLC
Three independent results, because this is the part usually argued by assertion:
| DORA, 2025 | AI adoption shows a positive relationship with throughput and a negative relationship with delivery stability. If you measure only speed, you will not see the cost. |
| METR, 2025 | In a randomized trial, experienced developers took 19% longer with AI tools — while believing they had been 24% faster. Self-reported productivity is not evidence. |
| GitClear, 211M changed lines | Copy-pasted code rose from 8.3% to 12.3% of changed lines; lines associated with refactoring fell from 25% to under 10%. Maintainability is the dimension most at risk, and it is invisible to a pass rate. |
Taken together those say something specific: generating code faster does not generate confidence faster, and the quality signal most likely to degrade is the one least likely to be on your dashboard. That is the argument for an independent verification layer — not because a vendor says so, but because the measured failure mode is exactly the one that speed metrics hide. What to do about it is in What to Measure: QE Metrics and KPIs with Studio.
How to use this
- Before signing anything, look up the terms in the contract. If a capability claim rests on a word this glossary flags as contested, ask the vendor for their specific definition in writing.
- When writing a test strategy, borrow the precise definitions rather than the marketing ones. The templates in Test Strategy Templates You Can Use Today already use them.
- When onboarding people, the four pages are short enough to read in an afternoon and will save weeks of people talking past each other.
Every definition links to a primary source. If you find one that is wrong, out of date, or that we have taken a side on when we should not have, raise a ticket — this page is meant to be corrected.