The single most expensive mistake in test automation is not a bad tool. It is automating the wrong things, discovering it six months later, and concluding that automation does not work.
The six questions that decide it
Score every candidate against these. A test that scores well on four or more is worth building. A test that fails the first two is almost never worth building, no matter how important the feature feels.
1. How often will it run?
Automation is a fixed cost paid once and recovered per run. A test that runs on every commit recovers its cost in days. A test that runs once a quarter may never recover it at all.
Good: regression suites, smoke tests, anything gating a release. Poor: one-off validation for a feature being retired next quarter.
This is the criterion Studio changes the most. When building a test is a plain-language description rather than a day of selector work, the run-frequency bar drops sharply — tests that were never worth scripting become worth describing. It does not drop to zero, though. Ask the question anyway.
2. Is the outcome deterministic?
Can you state, before the run, exactly what a pass looks like? If the answer involves "it depends" or "someone would need to look", you have a candidate for exploratory testing, not automation.
Good: a price displays as $966.00; an order confirmation shows an order number;
a permission-denied page appears for a user without the role.
Poor: "the layout looks right"; "the recommendations are relevant"; anything whose
expected value changes daily without a rule you can express.
Non-determinism is the root of almost all flake. Published analyses of open-source projects attribute roughly 45% of flaky tests to asynchronous waits and around 20% to concurrency — both are determinism problems that were present before a line of test code existed.
3. What does it cost if this breaks silently?
This is the business question, and it is the one that should dominate. Rank candidates by what the failure costs, not by how easy they are to build.
The classic mistake is to automate what is easy first — the settings page, the About page, the form validation — and arrive at a suite of 300 green tests that would not notice if checkout was broken.
Ask: if this broke at 2am on a Friday and nobody noticed until Monday, what would it have cost? Revenue? A compliance breach? A support queue? Reputation? If the answer is "very little", it is not a priority however cheap it is to build.
4. Is the behavior stable, even if the interface is not?
Note the distinction carefully, because it is where most teams misjudge.
A feature still under active design — where the rules change weekly — is a poor candidate. The test has to be rewritten every sprint because the thing it asserts keeps changing.
A feature whose rules are settled but whose markup changes constantly used to be an equally poor candidate, because every redesign broke every selector. That is exactly the constraint Studio removes: tests describe intent and elements are found visually and contextually, so a redesign that keeps the behavior does not break the test. See Self-Healing and Timing: Controlling How Studio Recovers.
The rule: automate settled behavior, not settled markup.
5. Can you control the data?
A test that depends on a record somebody else might change, a nightly refresh, or a customer account that gets cleaned up is not really a test. It is a scheduled source of false alarms.
Good candidates either create what they need, use data that is guaranteed stable, or take their values from a named variable set you control. In Studio that means an execution preset — see Test Data: Execution Presets, Variables and Secrets.
If a candidate needs data you cannot control, fix the data problem first. Automating on top of it produces a red suite that teaches people to ignore red.
6. Is it independent?
Can it run on its own, in any order, in parallel with everything else? A test that only passes when test 14 ran before it is a chain, and chains fail in ways that are miserable to diagnose. Roughly 12% of flaky tests in published analyses come from test-order dependency.
One test, one question. If a red result would leave you unsure which part failed, split it.
The scoring sheet
Score each candidate 1–3 on the six questions and total it. It takes two minutes per candidate and stops the loudest voice in the room deciding the backlog.
| 15–18 | Build it now. This is the core of your suite. |
| 11–14 | Build it, after the ones above. Note which criterion is weak and whether you can fix it — a data problem is usually fixable, a determinism problem sometimes is not. |
| 7–10 | Park it. Revisit when the feature settles or the data problem is solved. |
| Under 7 | Do not automate it. Say so out loud, and say why, so it does not come back next quarter. |
The order to build in
Candidate quality tells you whether. This tells you when.
- The money flow. Whatever a customer does that results in you being paid. One end-to-end test through it, with a real assertion at the end.
- The flows you have broken before. Go through the last quarter of escaped defects and write a test for each. This is the highest-yield list you will ever have, and it is already written down.
- Permission boundaries. Not just "can the admin do it" but "can the non-admin not do it". Negative cases are where real incidents live.
- The long, boring, repeated checks. The ones your team does by hand every release and quietly resents. Immediate, visible, morale-improving wins.
- Everything else, in descending order of the cost of silent breakage.
What to leave to a human, on purpose
Saying this explicitly is a mark of a mature program, not a gap in it.
- Exploratory testing. Nothing automated will find the problem nobody thought to describe. Protect the time for it.
- Subjective judgment. Is this readable? Is the tone right? Does this error message help? A person decides.
- Anything requiring a physical action — a card reader, a scanner, a real phone call.
- One-time migration checks that will never run again.
- Behavior still being designed. Come back when it settles. Writing the test early does not make the design settle faster.
The candidate-selection habit that beats all of this
For every defect that escapes to production, ask one question: could a test have caught this? If the answer is yes, write that test before the incident is closed.
That single habit builds a better suite than any planning exercise, because it is driven by evidence of what actually breaks in your system rather than by a guess about what might. It also produces, for free, exactly the number you need to justify the program — see What to Measure: QE Metrics and KPIs with Studio.
Where to go next
- Choosing What to Automate First — the sequencing plan for your first month.
- Writing Prompts That Produce Durable Tests — once you have picked a candidate, how to describe it.
- Keeping a Suite Green: Flake, Tiers and Who Owns Red — how to stop a good suite rotting.