The vocabulary of agents themselves — how they are built, what they remember, how they connect to your systems, and the named ways they fail.
A
Agentic system
An umbrella term covering both workflows, where models and tools are orchestrated through predefined code paths, and agents, where the model controls sequencing.
Why it matters for testing: Workflows are testable with conventional path coverage. Agents are not. Establish which you actually have before designing a strategy.
Usage is contested. The popular phrase "agentic workflow" is near-contradictory in this taxonomy, and three incompatible senses of "workflow" circulate.
AI agent
also: autonomous agent
A system that uses a model to direct its own process and tool use toward a goal, rather than following a path fixed in code.
Why it matters for testing: The control flow is chosen at runtime, so there is no fixed path to enumerate. Test design shifts from paths to outcomes and invariants.
Usage is contested. No cross-vendor agreement. Some definitions are broad enough to include ordinary rule-executing software.
Autonomy levels
A ranking of how much independent action an agent takes.
Why it matters for testing: There is no shared scale, so any "Level 3 autonomy" claim in a requirement or a contract needs its own local definition before you can test against it.
Usage is contested. No standards body or major vendor publishes an SAE-style scale. The best-known academic proposal defines five levels by the user's role: operator, collaborator, consultant, approver, observer.
C
Cascading failures
A false signal propagating through automated pipelines with escalating impact.
Why it matters for testing: An agent's output is often another agent's input, so a single wrong early step amplifies. Test error propagation across agent boundaries, not just within one agent.
Chain of thought
also: CoT
A series of intermediate reasoning steps emitted before an answer.
Why it matters for testing: Reasoning tokens are billable output and consume the context window — they are tracked separately in telemetry for exactly that reason.
Checkpointing
also: rollback, restore point, time travel
Persisting agent state so a run can be resumed, retried or branched from a prior point.
Why it matters for testing: The enabler for both error recovery and approval gates. Worth asserting on its limits: some implementations do not track files changed by shell commands, so it is not a general undo.
Compaction
also: auto-compact
Replacing older conversation turns with a model-written summary to keep a long run inside the context window.
Why it matters for testing: Compaction is lossy and non-deterministic. A bug that only appears after compaction is a real and easily missed class — run tests long enough to trigger it.
Usage is contested. The same word now names several different operations even within one vendor's surfaces, and other frameworks call it summarization.
Context engineering
Curating and maintaining the optimal set of tokens during inference — as against prompt engineering, which concerns wording. A common four-part split: write, select, compress, isolate.
Why it matters for testing: What is in the window at step N is the real input under test. Reproducing a failure means reproducing the assembled context, not just the user prompt.
Context poisoning
Two distinct senses. Accidental: a hallucination or error enters the context and is then repeatedly referenced. Adversarial: an attacker plants content in memory that reshapes behavior long after the initial interaction.
Why it matters for testing: Both are persistence failures — a single bad datum keeps producing wrong behavior after its cause is gone. With long-term memory, state must be reset between tests or one failure contaminates the suite.
Usage is contested. Two senses share the word and need different controls; say which you mean.
Context rot
Degradation of accuracy and recall as input length grows: models do not use their context uniformly, and reliability falls as the input gets longer even on simple tasks. The narrower lost in the middle effect is positional.
Why it matters for testing: A long-running agent's reliability decays over its own run. Test at realistic context occupancy, not on short synthetic cases.
Context window
Everything the model can reference when generating a response: system prompt, message history, tool definitions, tool results, images and the generated output itself.
Why it matters for testing: Tool definitions consume the window before the agent does anything, so adding connections silently reduces headroom.
D
Determinism and stochasticity
Agents are non-deterministic between runs even with identical prompts. Temperature zero is not sufficient: serving-side batch size varies with load and the underlying kernels are not batch-invariant.
Why it matters for testing: The most commonly mis-stated fact in agent QA. Greedy decoding removes sampling randomness but not serving randomness, so flaky-test triage cannot assume a fixed seed gives a fixed run.
E
Episodic memory
Memory of past agent actions, as against semantic memory (facts) and procedural memory (instructions).
Why it matters for testing: If the agent recalls its own past actions, a bad past action persists as a future bias. Regression tests must cover a populated memory, not only an empty one.
Usage is contested. Borrowed from cognitive psychology as an analogy. No settled definition or benchmark for agents.
Evaluator-optimizer
also: critic loop
One model call generates a response while another provides evaluation and feedback, in a loop.
Why it matters for testing: The nearest named vendor pattern to an automated QA gate inside the agent. Its evaluator needs its own validation, or you have an untested oracle.
Usage is contested. The common phrase "generator-critic dyad" has no primary source; cite evaluator-optimizer.
G
Goal drift
An agent deviating from its original objective over the course of a single long run, under context pressure.
Why it matters for testing: Not a synonym for goal misgeneralization: drift is degradation over time, misgeneralization is an objective that was wrong from the start. Test long-horizon runs and assert the original objective still holds at step N.
Usage is contested. An emerging term from a lab technical report rather than a settled one.
Goal misgeneralization
An agent retains its capabilities out of distribution but pursues the wrong goal — one that scored well in training and scores badly in novel situations.
Why it matters for testing: Competence is preserved, so the agent looks like it is working. Only out-of-distribution cases expose it, which is an argument for adversarial and novel test data.
H
Handoff
A transfer of control from one agent to another, exposed to the model as an ordinary tool call.
Why it matters for testing: Because handoffs are tool calls they appear in the trajectory and can be asserted on — and mis-routing is a distinct, testable failure class.
Usage is contested. Frameworks disagree: in one SDK a handoff is a one-way transfer, in another it is delegate-and-return. Not interchangeable.
Human in the loop
also: HITL, approval gate
A person able to approve, edit, reject or respond to an agent's proposed action before it executes. The MCP spec says there should always be a human able to deny a tool invocation.
Why it matters for testing: Requires checkpointing — approval and persistence are one architectural concern. The counter-risk to test for is approval fatigue: a gate that fires constantly stops being a real control.
I
Idempotency of tool calls
Whether calling a tool repeatedly with the same arguments has no additional effect.
Why it matters for testing: Agents retry. A non-idempotent tool — create order, send email, post a transaction — will eventually be called twice. Double execution is a test case, not an edge case.
L
Lethal trifecta
The three capabilities whose combination makes prompt injection catastrophic: access to private data, exposure to untrusted content, and the ability to communicate externally.
Why it matters for testing: The most useful quick risk triage for an agent's tool set. Enumerate the three and check whether any single configuration holds all of them — removing one breaks the chain.
Loop detection
Noticing that an agent is repeating itself.
Why it matters for testing: Do not assume the platform detects a loop — frameworks implement budget exhaustion, not semantic repetition detection. If you need "same tool, same arguments, N times", build and test it yourself.
Usage is contested. True repetition detection is not a standardized feature in any major framework.
M
MAST
also: Multi-Agent System Failure Taxonomy
An empirically derived taxonomy of 14 failure modes in three categories — system design, inter-agent misalignment, and task verification — built from expert-annotated traces across seven frameworks.
Why it matters for testing: The only evidence-based failure checklist for multi-agent systems. Usable directly as a review rubric.
MCP host, client and server
The host is the application that initiates connections and enforces consent. A client is a connector inside the host with a one-to-one relationship to one server, which exposes resources, tools and prompts.
Why it matters for testing: A key invariant to verify: a server should not be able to read the whole conversation, nor see into other servers.
Model Context Protocol
also: MCP
An open protocol for integration between AI applications and external data sources and tools, using JSON-RPC.
Why it matters for testing: The current revision is stateless — every request is self-contained. If your tests assume an initialize handshake and connection-scoped sessions, they target a superseded revision.
O
Observability and tracing of agent runs
Structured telemetry for agent execution. The vendor-neutral vocabulary is OpenTelemetry's GenAI semantic conventions, which define spans for agent invocation, planning and tool execution.
Why it matters for testing: The practical prerequisite for debugging non-deterministic runs. Note the conventions are still at development status, so attribute names can change.
Orchestrator-workers
also: manager pattern, lead agent and subagents
A central model breaks a task down, delegates to worker models, and synthesizes their results.
Why it matters for testing: Multi-agent systems have been measured using roughly 15× the tokens of chat, with token consumption explaining about 80% of performance variance. Cost is a first-class test dimension here.
P
Plan-and-execute
An architecture with a planner that produces a multi-step plan and executors that carry out each step.
Why it matters for testing: The plan is a separate, inspectable artifact. You can assert on plan quality before any step executes — the cheapest place to catch a misunderstood task.
Prompt caching
Resuming from a cached prompt prefix to cut latency and cost.
Why it matters for testing: Commonly miscategorized as memory. Caching changes what you pay for those tokens, not whether they still count toward the context window.
R
ReAct
also: reason-and-act
Prompting that interleaves reasoning traces with actions, so reasoning can revise plans while actions gather information from the environment.
Why it matters for testing: The reasoning trace is an artifact you can assert against, and is often where a wrong tool choice becomes explicable.
Reflection
also: Reflexion
An agent critiques its own output and retries. Reflexion specifically keeps that reflection in an episodic memory buffer across attempts — reinforcement by language rather than by weight updates.
Why it matters for testing: Reflection multiplies calls per task, so it interacts directly with step and cost budgets.
Reliability versus capability
Capability is average success; reliability is success every time. Measured with pass^k — the probability of succeeding across k repeated runs.
Why it matters for testing: The single most useful distinction for agent QA. Recent capability gains have yielded only small improvements in reliability, so a better model is not automatically a more reliable agent.
Replay
In current tooling, re-execution from a checkpoint — not deterministic reproduction. Steps after the checkpoint re-execute, including model calls, and may produce different results.
Why it matters for testing: Combined with serving non-determinism, deterministic replay of an agent run is not achievable with current tooling. Any test strategy premised on exact replay needs rethinking.
Usage is contested. The word strongly implies reproduction, and is routinely read that way.
Retrieval-augmented generation
also: RAG
Originally a specific architecture combining parametric and non-parametric memory with a jointly trained retriever and generator. Industry usage has broadened it to any retrieve-then-prompt pipeline over a frozen model.
Why it matters for testing: Retrieval quality and generation quality are separately testable and separately fail. Measure recall independently of answer correctness.
Usage is contested. The largest terminology drift in this glossary — an architecture in the paper, a pipeline pattern in current vendor docs.
Reward hacking
Finding a solution that formally maximizes the stated objective while perverting its intent. Obfuscated reward hacking is the case where the agent also learns to hide the fact in its reasoning trace.
Why it matters for testing: The reason a passing metric is not evidence of a correct solution. Watch specifically for agents that edit or special-case tests rather than fixing code.
Runaway agent
also: rogue agent
Colloquial terms for an agent taking unauthorized or unbounded action.
Why it matters for testing: The testable substance is bounded consumption, step caps, and a kill switch you have actually exercised — not the label.
Usage is contested. Informal. The nearest formal category covers misalignment, concealment and self-directed action; vendor guidance says runaway processes, not runaway agents.
S
Sandboxing
Operating-system enforced filesystem and network isolation for agent-executed commands and their child processes, so the boundary holds regardless of what the model decides to run.
Why it matters for testing: The primary containment control for an agent with shell access. The MCP spec adds that hosts should run server commands sandboxed with minimal privileges, since local servers otherwise run with full client privileges.
Self-correction
An agent revising its own answer. The critical split is intrinsic self-correction, with no external signal, against correction using real feedback such as a failing test, a type error or a tool error.
Why it matters for testing: The evidence for intrinsic self-correction is negative — models struggle to self-correct without external feedback, and performance sometimes degrades. Give the agent a real signal rather than asking it to re-check itself.
Usage is contested. Vendor material frequently blurs the two cases, which have opposite evidence.
Short-term and long-term memory
Short-term memory is scoped to a thread or session and tracks the ongoing conversation. Long-term memory stores user- or application-level data across sessions and is shared between them.
Why it matters for testing: Long-term memory makes runs non-independent. Test isolation now requires resetting the store, or tests contaminate each other.
Usage is contested. A framework convention rather than a formal result; different frameworks scope it differently.
Skills
also: Agent Skills
Organized folders of instructions, scripts and resources an agent discovers and loads on demand, using progressive disclosure: name and description first, full contents when judged relevant, bundled files only as needed.
Why it matters for testing: Only the name and description are always in context, so a skill that fails to trigger is a different failure from a skill whose contents are wrong. Test discovery and execution separately.
Specification gaming
Behavior that satisfies the literal specification of an objective without achieving the intended outcome.
Why it matters for testing: Names the gap between your acceptance criteria and your intent. It is a specification defect, not an algorithm defect — the fix is in the criteria.
Step budget
also: max iterations, recursion limit
A hard cap on agent-loop iterations, after which the run is stopped.
Why it matters for testing: A core safety control — but the units differ between frameworks (model turns, graph supersteps, event-loop cycles), so the numbers are not comparable.
Subagent
A delegated agent instance with its own context window and task, spawned by a lead or orchestrator agent.
Why it matters for testing: The separate context window is both the main benefit and the main hazard: a subagent cannot see what the parent knows unless it is passed explicitly, which is a common silent failure.
T
Tool annotations
also: risk hints
Optional behavioral hints on a tool: read-only, destructive (defaults to true), idempotent (defaults to false), and open-world.
Why it matters for testing: These are hints, not guarantees — there is no enforcement, and the spec says clients must treat annotations from untrusted servers as untrusted. Verify idempotency yourself.
Tool hallucination
Agent actions unfaithful to the instructions, the execution history or the observed environment — commonly, calling a tool that does not exist or inventing arguments no schema declares.
Why it matters for testing: Both common cases are catchable by strict input-schema validation, which makes this one of the cheaper agent failure modes to defend against.
Usage is contested. No single canonical definition; some standards bodies prefer "confabulation" for the whole family.
Tool schema
A tool definition's name, description, input schema and optional output schema, expressed as JSON Schema.
Why it matters for testing: An output schema gives you a machine-checkable contract on tool results — the most direct schema-validation assertion available in an agent stack.
Tool use
also: function calling, tool calling
The model returns a structured call naming a tool and its arguments; your application executes it and returns the result. The model emits the request — it does not run the code.
Why it matters for testing: The model chooses which tool and what arguments. Both are outputs under test, independent of whether the tool itself works.
Trajectory
also: agent path, tool-call sequence
The ordered list of steps an agent took before returning to the user — the tool calls and intermediate actions, as against the final response.
Why it matters for testing: The core agent-specific test artifact. Two agents can produce the same answer by very different routes; trajectory assertions catch the one that got there by luck, or unsafely.
Usage is contested. A vendor-tooling term rather than one with a canonical academic definition. Some benchmarks compare end state instead.
Tripwire
A guardrail check that, when it fails, immediately halts agent execution rather than merely flagging.
Why it matters for testing: Input tripwires also save cost by rejecting before the expensive model runs. Test the guardrail itself — it is code with its own false-positive rate.
U
Unbounded consumption
also: cost budget, runaway spend
Allowing excessive and uncontrolled inference, leading to denial of service, economic loss or service degradation. Mitigated with rate limits, quotas, timeouts and throttling.
Why it matters for testing: The formal vocabulary for "the agent burned $400 overnight". Assert on tokens and cost per task, not only on pass or fail.
49 terms. Spotted something wrong or missing? Raise a ticket — this page is meant to be corrected.