Scheduling an Orchestration: The Six Cadences and What Each One Costs

An orchestration's schedule is the single setting that decides what it costs and whether anyone still trusts it in six months. Everything else about a suite — which tests, what order, who gets the email — matters less than how often it runs.

There are six options. This page covers all of them, including the one that is not there.

The six schedule options, and the day picker that appears when you choose Daily

Where the setting lives

The Schedule field is the fourth field in the New Orchestration panel, and it is on an existing orchestration under Actions → Edit. Changing it does not disturb anything else about the suite.

Whatever you pick, Studio prints a one-line description next to the field. For the default that line reads "Runs only when triggered manually." — worth reading, because it is the only confirmation you get that you have not accidentally armed a timer.

The six options

On Demand The default, and the right answer for a brand new suite. Nothing happens until somebody presses Run, so the orchestration costs nothing while you are still working out whether it is any good.
Hourly Twenty-four runs a day. Almost nobody needs this, and the people who do have a three-test smoke suite rather than a thirty-test regression pack. Before you pick it, work out what you would do differently on being told about a failure at 3am versus at 9am.
Every 4 Hours Six a day. A sensible ceiling for a suite that gates deploys in a team shipping several times a day.
Every 12 Hours Twice a day. In practice this is "before the working day and after it", which suits a team spread across two time zones.
Daily Once a day, at a time you pick, on the days you pick. This is where most suites should end up, and it is the option with the most hidden flexibility — see below.
Monthly Once a month. Right for a long accessibility, localization or compliance pass that nobody was ever going to read weekly.

There is no Weekly, and that is fine

This catches people out, because weekly is a perfectly reasonable cadence for a full regression pack. The way to get it is Daily.

Choosing Daily adds two controls that are invisible until you do:

  • at — a time field, defaulting to 12:00 AM.
  • on — a day picker holding Monday through Sunday, and it takes as many as you like.

So a weekly job is Daily, at 02:00, on Sunday. A weekdays-only job is Daily, at 06:00, on Monday through Friday. A twice-weekly job is Daily on Tuesday and Thursday. The naming is a little odd, but the control is more capable than a Weekly option would have been.

Leave the day picker empty and Daily means every day
If you never open the day list, the job runs seven days a week. That is usually what you want from a smoke suite and almost never what you want from a two-hour regression pack.

What each cadence actually costs

An orchestration consumes credits for the tests it runs, exactly as if you had run them one at a time. Grouping them is organizational; it is neither a discount nor a surcharge. What the schedule changes is the multiplier.

On Demand However many times somebody presses the button.
Hourly About 720 runs a month.
Every 4 Hours About 180 runs a month.
Every 12 Hours About 60 runs a month.
Daily (every day) About 30 runs a month; five weekdays only is about 22; one day a week is about 4.
Monthly 1 run a month.

Multiply by the number of tests in the suite and you have the monthly cost of that orchestration. A twelve-test suite on Hourly is doing 8,640 test runs a month; the same suite on weekdays at 6am is doing 264. Both are defensible. Only one of them is defensible by accident. See How Credits Work.

Choosing a cadence you will still want in six months

The honest test is not "how often would we like to know?" — everyone answers that with "constantly". It is this:

  • Who reads the result, and how soon? If the answer is "we look on Monday", an hourly schedule is generating 167 results a week that nobody opens.
  • What does a failure block? A suite that gates a deploy earns a frequent schedule. A suite that informs a trend does not.
  • Is it stable? A flaky suite on a frequent schedule trains the team to ignore red, and that habit is much harder to undo than a schedule is to change. Put it on On Demand until the flake is fixed — see Keeping a Suite Green: Flake, Tiers and Who Owns Red.

A practical default for a team getting started: smoke on Every 4 Hours, core on Daily at whatever hour is quiet for your application, full on Daily with one day ticked.

Turning a schedule off again

Set it back to On Demand. The orchestration, its tests and its whole run history stay exactly as they are; it simply stops firing on its own. This is the right move when a suite goes flaky, when the application is mid-migration, or when somebody leaves and nobody has picked up their alerts.

The Schedule filter on the orchestrations list — which offers On-demand and Scheduled — is the fastest way to audit what is firing without you. Doing that once a quarter is a habit worth forming. See Managing Orchestrations: Filters, Bulk Actions and Run History.

Where to go next

Download this article
A PDF of this page, for reading offline, printing, or passing to somebody who does not have a Studio account yet.
Scheduling an Orchestration: The Six Cadences and What Each One Costs (PDF, 251 KB)