Most suites do not fail all at once. They erode: a test that fails intermittently gets rerun instead of fixed, then another, then the team learns that red usually means nothing, and within a quarter nobody reads the result at all. The suite still runs. It just no longer does anything.
Tier your suite, and mean it
A single undifferentiated pile of tests cannot be trusted, because a failure in it carries no information about how urgent it is. Three tiers is enough.
|
Smoke Under 10 minutes |
A handful of tests that answer "is this deployment fundamentally alive?" Runs on every deploy. Blocks the release. Must be fast, and must never be flaky — a gate that fails at random gets bypassed within a month, and then nothing is gated. |
|
Critical Runs daily |
Every revenue-critical journey and permission boundary. Usually ten to twenty tests. A red result here is an incident, not a chore. |
|
Regression Runs nightly or per release |
Everything else, including one test per escaped defect. Failures here are triaged the next working day rather than paged on. |
In Studio, tiers are orchestrations. Build one per tier, schedule them differently, and tag the underlying tests so you can tell at a glance which tier a test belongs to. See Exercise 6 - Group Tests Into an Orchestration.
The rule that keeps the tiers honest: a test may only be promoted upward after it has passed reliably where it is. Nothing enters the smoke tier because it feels important. It enters because it has earned it.
Flake is a defect, not weather
This is the belief that decides whether your suite survives.
Flake is not background noise you learn to live with. It is a defect — in the test, in the data, or in the application — that happens to be intermittent. Google has reported that around 16% of its tests show some flakiness, and that the large majority of pass-to-fail transitions involve a flaky test. Industry studies put real money on it: developers in one industrial case study spent over 1% of their time simply repairing flaky tests.
The costs people quote are the small part. The real cost is the habit it teaches: that red sometimes means nothing. Once a team believes that, every genuine failure has to compete for attention with the noise.
The four causes, and what to do
|
Timing ~45% of flaky tests in published analyses |
A step ran before the page was ready. Never fix this with a fixed wait. Describe the state you are waiting for, raise the Missing element timeout, or move the Timing model toward Conservative — see Self-Healing and Timing. |
| Data | The record the test needed was changed or cleaned up. Make the test create what it needs, or pin the values in an execution preset you control. |
|
Order dependency ~12% of flaky tests |
The test only passes when another ran first. Split the dependency out or make the test set up its own state. Tests must be independent. |
| Environment | A shared staging environment being redeployed mid-run, a third-party sandbox that rate-limits, a cookie banner that appears for some sessions. Fix it at project level once rather than in every test. |
The rule for handling a flaky test
When a test fails intermittently, you have exactly three options. "Rerun it and move on" is not one of them.
- Fix the cause — the right answer almost always.
- Demote it out of the tier that gates anything, with a dated note saying why, until it is fixed.
- Delete it. A test nobody trusts and nobody will fix is worse than no test, because it consumes runs, attention and credibility.
Quarantine with no date is how flake becomes permanent. If you demote something, put the date on it.
Somebody owns red
This is the single decision that most strongly predicts whether a suite is still being read in six months, and it has nothing to do with tooling.
For every scheduled orchestration, write down:
- Who looks at the result — a named person, not "the team".
- When — within an hour for smoke, same day for critical, next working day for regression.
- What they do when it is red — the actual first action.
- What "done" means — either the application was fixed, or the test was fixed, or the test was demoted with a date. Never "it passed on the rerun".
A scheduled suite with no named reader manufactures confidence. It is worse than having no suite, because people believe it.
Keep the suite pruned
Suites accumulate. Once a quarter, spend an hour on this:
- Archive tests for retired features. They cost credits on every scheduled run and prove nothing.
- Find the tests that never fail. Filter the test list by last run and last updated. A test that has passed for a year without the feature changing may be asserting nothing — break it on purpose and see whether it notices.
- Find the tests that always fail. Somebody has been ignoring them. Fix, demote or delete.
- Check the smoke tier is still under ten minutes. Tiers drift upward as people add "just one more".
- Look for duplicated flows. A login repeated in twenty tests should be a reusable component — the Add component control in the steps pane toolbar. Cheaper to maintain and cheaper to run.
The numbers that tell you it is working
Three, checked weekly in Reports:
- Warning count relative to failures. If warnings are comparable to failures, nobody is triaging them.
- Smoke suite duration. Watch it creep.
- Repair frequency. The History panel on a test shows every change with its author. A test accumulating repeated edits is telling you something about the test or about the feature.
Full detail in What to Measure: QE Metrics and KPIs with Studio.
Why this gets easier with Studio, and what still needs you
Two of the four flake causes above — the ones that dominate selector-based suites — are substantially reduced by the product itself. Tests describe intent rather than structure, so a redesign that preserves behavior does not produce a wave of red. Self-healing recovers from drift rather than failing on it. The maintenance share that makes traditional suites stop growing is exactly what the platform targets.
The other two are yours. No tool can decide that your staging environment should stop being redeployed mid-run, or that a test needs its own data rather than borrowing someone else's, or that a named person reads the smoke result every morning. Those are decisions, and they are the ones that determine whether the suite is still trusted next year.
Where to go next
- Self-Healing and Timing: Controlling How Studio Recovers — the settings behind the timing tier.
- Diagnosing a Failed Test in Functionize Studio — the skill every person on the rota needs.
- Reviewing AI-Generated Tests: What a Human Still Has to Check — stopping weak tests entering the suite in the first place.