Exercise 3 - Read a Test Run and Its Evidence

A run keeps far more than a pass or a fail. This exercise teaches you to read one, so that when something goes wrong later you already know where to look. About ten minutes.

You need a test that has run at least once.

Step 1 — Understand the numbering

Open your test and look at the steps pane:

The steps pane showing instructions and their generated actions

The numbers are doing real work:

  • Whole numbers (1, 2, 3) are the instructions you wrote.
  • Decimals (1.1, 1.2 …) are the actions Studio generated to carry them out.

Expand an instruction and count. Opening a product is typically several actions: load the home page, click through the category menu, load the category, click the product, load the product page. When something fails later, the decimal tells you which action broke and the whole number tells you which of your intentions it belonged to.

Step 2 — Read the summary

Read the agent's summary on the left and take two things from it every time:

  • How many steps passed, out of how many.
  • Which browser and version it ran on.

Note the browser version down. "It passes for me" and "it fails on the schedule" are very often the same test on two different browser builds, and checking that first will save you an afternoon.

Step 3 — Look at the evidence

Click any step. The right-hand pane shows the page as it was at that moment:

The page as it was at the selected step

This is a real screenshot from the run, not a reconstruction. If you ever doubt that a step did what you think it did, this settles it.

Step 4 — Learn the Data tab

This is the part worth remembering. Underneath the screenshot, Data shows the values that step read, in three columns:

The Data tab comparing Previous Successful Run, Current and Generated

Previous Successful Run What this value was the last time the test passed.
Current What it is in this run.
Generated What it was when the test was first created.

Read them together and most failures explain themselves:

  • Current differs from Previous Successful Run — your application changed, and the difference is the change.
  • They match but the step still failed — the application is fine and the expectation moved.
  • Previous Successful Run says there is no data — this test has never passed in this browser.

Do this now, on a passing test, while there is no pressure. It is much harder to learn a new screen while something is broken.

Step 5 — Glance at the other tabs

Beside Data are Log, Extension, Variables and Components Model. You will not need them often. The two worth knowing:

  • Log — the raw execution log for that step.
  • Variables — values carried from one step to another. If a later step uses something an earlier step captured, check here that it was captured correctly.

Checkpoint

Without looking back at this page, answer these about your own test:

  1. How many actions did your first instruction expand into?
  2. Which browser and version did your last run use?
  3. For one step, what does the Data tab show under Current?

What you learned

A run is a report, not a verdict. The failing step tells you where, the screenshot tells you what the page looked like, and the Data comparison usually tells you why.

Next: Exercise 4 — Change a test by asking.