A printable two-page handout of everything on this page: a week-one baseline worksheet to fill in, the seven KPIs with their targets and sources, the reporting rhythm, first-quarter targets and the three numbers that justify the spend.
Download the PDF (129 KB)
If you cannot say what improved, you cannot defend the investment. This page covers what a quality engineering team should measure when adopting Studio, how to capture a baseline before you start, and exactly where each number comes from in the product.
Capture these before you build a single test
None of these come from Studio. They come from your existing process, and they are the numbers that matter to everyone outside QE.
| Escaped defects | How many issues reached production in the last quarter. Your incident or ticket system. |
| Manual regression hours | Person-hours spent hand-testing per release. Ask the team; they will know. |
| Release cadence | How often you ship, and how long a release is held for testing. |
| Test maintenance time | Of the hours spent on automation, what share goes on repairing existing tests rather than writing new ones. If you have an existing suite, this is usually the most damning number you have. |
| Critical journeys covered | List your revenue-critical flows and mark which are covered by automation today. Often a sobering list. |
Write them down with the date. Six months later this is what shows the difference.
The KPIs worth owning
1. Escaped defects — the north star
Everything else is a proxy for this. If escaped defects are not falling, a rising pass rate means nothing.
Target: a downward trend quarter on quarter. Source: your ticket system, tagged by whether automation could plausibly have caught it.
That tag is the useful part. For every escaped defect ask: could a test have caught this? If yes, write that test now. That single habit does more for coverage than any planning exercise.
2. Critical journey coverage
Not "percentage of the application" — percentage of the flows that would hurt if they broke silently.
Target: 100% of your critical list, which is usually ten to twenty flows. Source: your own list, checked against your Studio projects.
3. Time to feedback
How long from a change landing to knowing whether it broke something.
Source in Studio: the duration on an orchestration's run history. Our two-test demo suite runs in 1 minute 14 seconds; your smoke suite should be minutes, not hours. If it is not, it has too much in it.
4. Maintenance share — the one Studio targets directly
The share of automation effort spent fixing tests rather than adding coverage. In selector-based suites this commonly runs above half, which is why those suites stop growing.
Source in Studio: the History panel on a test shows every change with who made it and when. A test accumulating repeated edits is telling you something. Across the suite, falling repair frequency is the number that justifies the platform.
5. Flake rate
Tests that fail for reasons unrelated to the product. Flake is corrosive because it teaches people to ignore red.
Source in Studio: the Warning count in Reports is a good proxy — runs that completed with something worth looking at. If warnings are comparable to failures, nobody is triaging them.
Target: near zero, and treat any recurring flake as a defect in the test or the environment, not as background noise.
6. Pass rate — useful, but not on its own
Source in Studio: Reports gives an overall figure and a Pass Rate Breakdown by Project.
Read it with care. A project at 100% with one trivial test is worse than one at 85% with forty demanding ones — which is exactly why the table shows the test count beside the rate. And a falling pass rate often means someone added genuinely hard tests, which is good.
7. Mean time to diagnose
How long it takes to explain a red test. Rarely measured, and it is where teams actually lose their days.
Source: time it yourself for two weeks. With the Execution Error, the step screenshot and the Data comparison, this should be minutes. If it is not, people have not been shown how to read a failure.
Where the numbers live in Studio
| Reports | Execution Breakdown (passed / failed / warning, counts and percentages), Test Execution Trends per day, Pass Rate Breakdown by Project, a date range carried in the URL, and Export. |
| Orchestration run history | Duration and result per suite run — your time-to-feedback number. |
| Test History panel | Every run and every change, with author and timestamp — maintenance activity and run-duration drift. |
| The test list | Status, last updated, last run and browsers, filterable by tag — useful for spotting tests that have not run in months. |
One caveat printed on the Reports page itself, and worth respecting: "Data updates on a schedule and may not reflect the latest changes." Reports is for weekly and monthly shape, not for checking whether the fix you made two minutes ago worked. For that, open the run.
A reporting rhythm that works
Weekly, for the team. Open Reports on a seven-day range. Look at the trend, not the number. Sort the project table by pass rate and look at the bottom. Check whether warnings are growing. Pick one thing to fix.
Monthly, for leadership. A thirty-day range, exported. Report three things: escaped defects, critical journey coverage, and manual regression hours displaced. Those are the numbers non-QE people can act on — pass rate is an internal diagnostic.
Quarterly. Compare against the baseline you captured in week one. This is the conversation that renews budget.
Targets for a first quarter
Deliberately modest, because over-promising in month one is how programs get canceled in month six.
- 100% of critical journeys covered — usually ten to twenty tests.
- Smoke suite under ten minutes, running at least daily.
- Flake near zero — every recurring warning triaged, not tolerated.
- Maintenance below a quarter of automation effort.
- Every escaped defect gets a test, without exception.
Justifying the spend month over month
After you upgrade, someone will ask what you are getting for it. Answer with three numbers, in this order — and note that only the last one is about the tool.
- Hours displaced. Manual regression hours before, versus now. This is the easiest number for a finance conversation and usually the largest.
- Escaped defects prevented. Count the failures your suite caught before release that would previously have reached production. Tag them as you go; reconstructing them later is guesswork.
- Credit consumption, with its shape explained. Consumption is front-loaded: building a test costs roughly the same as fifty runs. Month one is mostly building and looks expensive; by month three you are mostly running, and the run-rate settles.
If you present credit usage without explaining that curve, a normal build-heavy first month reads as a cost overrun. Show the trend, not the snapshot.
The efficiency metric worth tracking: credits per test maintained per month. It should fall as your suite matures — because repairs cost a fraction of rebuilds, manual edits are free, and reusable components stop you regenerating the same login twenty times. A rising figure means people are regenerating tests instead of fixing them, which is a training issue rather than a pricing one.
How consumption works, and the habits that reduce it, are in How Credits Work.
Two ways to measure badly
Counting tests. "We have 400 tests" says nothing about whether you would catch a broken checkout. Report what is proven, not what was written.
Chasing a coverage percentage. It drives people to write cheap tests over valuable ones. Ask instead: if the whole suite passes, what do we now know is true?
See Choosing What to Automate First for building the critical list, and Reports for reading the screen itself.