An agent can write a test faster than you can read one. That asymmetry is the productivity gain, and it is also the risk: it is entirely possible to acquire four hundred tests in a week, none of which would notice if checkout broke.
The one question
Before the checklist, the question that catches most weak tests on its own:
If the thing this test is supposed to protect were broken, would this test go red?
If you cannot answer yes with confidence, nothing else on this page matters. The fastest way to find out for certain is Exercise 5 - Break a Test on Purpose, Then Diagnose It: change the expected value to something wrong and confirm the test fails. A test you have never seen fail is a test you have no evidence about.
The checklist
1. Does it assert something, and is the assertion specific?
Open the steps and find the verification. "The subtotal is displayed" passes when the subtotal is wrong. "The subtotal is $966.00", or "the subtotal equals the sum of the line prices", does not.
A generated test that walks a journey and ends without checking anything is the single most common weak output, because walking the journey is the part the agent found easy.
2. Is it asserting on intent rather than structure?
Check the agent did not quietly pin the test to something incidental — a row position, an element's index, a generated identifier. Those survive until the next redesign. Named things and displayed values survive longer.
3. Does it test one thing?
If a red result would leave you asking "which part failed?", it is too long. Long tests are also where generated output tends to drift: by step forty the agent is several inferences away from what you asked for.
4. Is there a negative case?
Positive paths get exercised by hand constantly. Negative ones almost never do, and that is where incidents live. Does anything confirm that the guest is not asked to sign in, that a non-admin cannot open the screen, that no error banner appears?
If there is no negative assertion anywhere in the suite for a feature, the suite is incomplete regardless of its size.
5. Is the data controlled?
Does the test depend on a record someone else might change, or on whatever happened to be in staging this morning? Trace every value it relies on. Anything not controlled is a future false alarm with a date on it.
Secrets get their own check: nothing sensitive in a prompt or a step, ever. Credentials belong in an execution preset variable with Encrypt value switched on — see Test Data: Execution Presets, Variables and Secrets.
6. Does it match the requirement, or the implementation?
The one that needs a person, because it needs someone who knows what was actually asked for.
Read the test beside the acceptance criteria. Does it test what the story asked for? Or has it quietly encoded what the application currently happens to do? An agent with access to the implementation will test the implementation — which is why the testing session should not have that access at all. See The Independent Tester Principle.
7. Would the person who wrote the requirement recognize it?
Studio tests are in plain language, so you can actually run this check. Show the test to the product owner or analyst who wrote the story. If they read it and say "yes, that is what I meant", you have verified the thing that no amount of code review can verify.
This closes the loop back to the source of the requirement, and it catches specification misunderstandings while they are still cheap.
8. Is it fast enough for the tier it is going into?
A three-minute test is fine in regression and wrong in smoke. Check the duration on its first run before deciding where it belongs. See Keeping a Suite Green: Flake, Tiers and Who Owns Red.
9. Has it passed three times in a row?
Do not admit anything to a tier that gates a release until it has. Reliability is demonstrated, not assumed, and the cheapest moment to discover flake is before anyone depends on the test.
What to do when a test fails the review
Ask, in plain language, rather than rebuilding. "Add a check that the subtotal equals the sum of the line prices." "Also confirm the guest is never prompted to sign in." "Split this into two tests, one for the cart and one for checkout."
Asking for a change costs a fraction of regenerating the test, and small edits you make by hand cost nothing at all. See How Credits Work and Exercise 4 - Change a Test by Asking.
A review rhythm that scales
| Every new test | The nine checks above. Ninety seconds. No exceptions for tests entering the smoke or critical tier. |
| Weekly | Pick two existing tests at random and break them on purpose. Confirm they go red for the right reason. This is the only way to find tests that quietly stopped asserting. |
| Per release | Read the list of what the suite covers and ask the closing question: if all of this passes, what do we now know is true? The gaps are more informative than the coverage. |
| Quarterly | Prune. Archive retired features, delete tests nobody trusts, fold duplicated flows into components. |
The mindset
You are not reviewing the agent's work the way you would review a junior colleague's, looking for mistakes to correct. You are asking a narrower and more useful question: does this test have teeth?
An agent will reliably produce a test that runs. Whether it produces a test that would catch something depends on how clearly you described what must be true — which is why Writing Prompts That Produce Durable Tests and this page are really the same discipline seen from two ends.
The review is also where the compounding happens. Every time you catch a missing oracle and ask for it, you learn something about how to phrase the next request, and the next batch needs less correcting. Teams that skip the review never get that feedback, and their four hundredth test is exactly as weak as their first.
Where to go next
- Exercise 5 - Break a Test on Purpose, Then Diagnose It — the single most valuable exercise in the series, and the practical form of check one.
- Writing Prompts That Produce Durable Tests — how to stop weak tests being generated at all.
- Setting Up an Adversarial Test Loop With Claude and the Functionize MCP Server — who should be writing the tests you are reviewing.