Search

Check Yourself: Twenty Questions on Testing With AI

Twenty questions, grouped into four rounds. Each one has a short answer behind it — click the question to see it, and a link to the article that explains it properly.

The four rounds of the self-test and what each one covers

Four rounds of five. Best run with a team, because the disagreements are the output.

This is not a certification and nothing is scored. It is a way to find out, in about twenty minutes, which parts of your team's model of testing are out of date — which is a much more useful thing to know than a percentage.

How to run this with a team
Read each question aloud and let people answer before anyone opens the card. The disagreements are the point — where two people on the same team answer differently, you have found something worth writing into your test strategy. Budget five minutes a round for discussion.

Round 1 — Does your suite prove anything?

Five questions about what a passing suite does and does not tell you. Most teams get question three wrong on the first pass, and it is the one that matters most.

1. Every test in our suite passes. What do we now know is true?
Only that the behaviors you wrote checks for still behave. If that list does not include "customers can do the thing we charge them for", a green suite is not evidence the product works. This is the single most useful question to ask of any suite, and it is uncomfortable the first time.
2. We have 80% coverage. Is that good?
The claim is malformed until you say 80% of what. Both ISTQB and ISO define coverage against a stated coverage item. Line coverage — the usual default — measures which lines ran, not which risks were checked. A suite can execute every line while asserting nothing.
3. We added 300 tests this quarter. Is that a result?
No, and it is the easiest number to move without improving anything — especially now that writing tests is cheap. Report what is proven, not what was written. The numbers that carry information are escaped defects and maintenance share.
4. A test fails intermittently. Someone calls it flaky. What are the options?
Exactly three: fix the cause, demote it out of any tier that gates something with a date, or delete it. Quarantine with no date is how flake becomes permanent — and "flaky" is where real race conditions get filed and forgotten, because neither ISTQB nor ISO defines the word.
Explained in full: Keeping a Suite Green
5. Our pass rate is 97%. Are we in good shape?
Unknown, because pass rate is confounded by the thing it is used to hide. If the suite retries automatically, that number measures your retry policy as much as your product. Publish it alongside first-run pass rate and flake rate.
Explained in full: Keeping a Suite Green

Round 2 — Working with an agent

Five questions about what changes when an AI agent is doing the building. These are the ones where intuition from pre-agent testing quietly misleads.

6. Our coding agent writes the code, runs the tests and reports green. Is that enough?
No, and this is the sharpest risk in the whole model. An agent that runs your suite treats green as its stopping condition, so a weak test stops merely failing to catch bad code and starts actively licensing it. The thing that verifies has to be independent of the thing that builds.
7. Why use a different model for the testing session?
Different training produces different blind spots. A reviewer that shares the builder's assumptions will agree with them confidently. The point of an adversarial loop is disagreement you can act on, and you only get that from genuine independence.
8. We set temperature to zero. Are our tests reproducible now?
No. Production endpoints are non-deterministic even at temperature zero, because server-side batch size varies with load. "It passed once" is not evidence. Plan for repeated trials and read the rate, not the run.
9. The agent says it can replay the run. Does that mean we can reproduce the failure?
Not in any current framework. Replay means re-execution from a checkpoint — steps after the checkpoint run again and may produce different results. Deterministic replay of an agent run is not achievable with today's tooling, so a test strategy premised on it needs rethinking.
Explained in full: Agentic Systems: Terms
10. A long agent run starts making careless mistakes near the end. Why?
Reliability decays as context fills — models do not use a long context uniformly, and accuracy and recall fall as input length grows, even on simple tasks. Test at realistic context occupancy rather than on short synthetic cases.
Explained in full: Agentic Systems: Terms

Round 3 — Strategy and scope

Five questions about deciding what to automate, and what not to. The wrong answer here costs more than any tooling decision.

11. Should we automate everything?
No, and saying so explicitly is a mark of a mature program. Exploratory testing, subjective judgment, anything needing a physical action, one-time migration checks, and behavior still being designed all stay manual on purpose. Write that list down.
12. How many critical journeys should we have?
Usually ten to twenty. If your list has sixty, you have listed features rather than journeys. The test: if this broke at 2am on a Friday and nobody noticed until Monday, what would it have cost?
13. What is the single most valuable page of a test strategy?
An honest baseline of what you test manually today, with the date on it. Six months later it is the only evidence you have of where you started — and it is the section teams are most tempted to skip.
14. Who should own a red result?
A named person per scheduled suite, with a timescale and a first action. This single decision predicts whether the suite is still being read in six months better than anything else about it.
Explained in full: Keeping a Suite Green
15. We are moving from SAP ECC to S/4HANA. Where is the automation case?
Not in the first pass of any level. It is in the repetition - the same scenarios executed by hand in SIT cycle 1, again in cycle 2, again in UAT, again at the next wave. And settle the access-route question early: a browser drives Fiori and SAP GUI for HTML, but not SAP GUI for Windows.

Round 4 — Measuring in an AI-DLC

Five questions on proving the program is working. If your team can answer these, your next steering meeting will go differently.

16. AI made our developers faster. How do we show it?
Carefully, because self-reported speed is not evidence. In a randomized trial, experienced developers took 19% longer with AI tools while believing they had been 24% faster — and the code quality was the same either way. Pair every survey number with a measured one.
17. We track DORA metrics already. Does anything change?
Two things. Add a stability measure if you only track throughput, because that is exactly where AI adoption shows a negative relationship. And check your definitions: DORA has retired MTTR in favor of failed deployment recovery time.
Explained in full: Frequently Asked Questions
18. Where did the time we saved on code generation go?
Into verification. DORA calls it the verification tax - time saved during generation is re-allocated to reviewing what was generated. Naming it is what lets you argue for an independent test layer instead of more reviewers.
19. A vendor claims a 70% reduction in test maintenance. How do we check it?
Ask for the definition, the denominator and the baseline, in writing. There is no standard definition of maintenance share — and ISTQB defines neither "flaky", "self-healing", "autonomous testing" nor "code coverage" either.
20. We have measured nothing so far. Where do we start?
Half an hour, this week, before building anything: escaped defects last quarter, manual regression hours per release, and which critical journeys are covered today — with the date written on it. Everything else can follow.

What to do with the ones you got wrong

Nothing, immediately. Note which round produced the most disagreement and take that one to the relevant page: What to Measure: QE Metrics and KPIs with Studio for round four, The Independent Tester Principle for round two, Test Strategy Templates You Can Use Today for round three.

If round one produced disagreement, start there regardless of what else you planned. A team that does not agree on what a passing suite proves will not agree on anything downstream of it either.