The questions people actually ask in their first weeks with Functionize Studio, grouped by when
they tend to come up. Each answer links to the article that covers it properly.
Open in Studio
The screens these answers refer to. Links open in a new tab.
Getting started
| Q |
What is Studio, in one sentence? |
| A |
An independent testing agent for your web application: you describe what should be true in plain
language, and it builds the test, runs it in a real browser, and keeps it working as your
application changes. See What Is Functionize Studio?
|
| Q |
Do I have to install anything? |
| A |
No. Tests run on our infrastructure in a real browser. There is nothing to install or maintain. |
| Q |
What should I do first? |
| A |
Work through Your First 30 Minutes with Functionize Studio. It deliberately starts on our
demo store rather than your application, so you learn what "working" looks like before you meet your
own environment's quirks. |
| Q |
Is there a practice application I can use? |
| A |
Yes, and no sign-in is needed:
https://ecom.functionizeapp.com/testsites/anaqa/index.php?route=common/home. The whole
exercise series uses it. |
Setup and access
| Q |
Studio cannot reach my application. What now? |
| A |
This is the most common first-week problem and it is almost always access rather than testing. Work
through Before You Point Studio at Your Own Application. If your app is not on the public
internet, stop and ask us rather than losing a week. |
| Q |
My app is behind a login prompt from the browser itself. |
| A |
That is HTTP basic auth: Project Settings → Access & Auth → Basic
authentication. |
| Q |
We use self-signed certificates on staging. |
| A |
Certificate handling → Allow invalid TLS certificates. |
| Q |
A cookie banner blocks every test. |
| A |
Do not add steps to dismiss it. Set the consent cookie once at project level under
Access & Auth → Cookies. |
| Q |
Should I test against production? |
| A |
Generally no — you will create real orders and send real emails. Use a stable QA or staging
environment. An unstable environment produces failures that teach your team to distrust the tool. |
Writing tests
| Q |
Why should I not describe CSS selectors? |
| A |
Because Studio finds elements visually and contextually. Describing intent is what lets the test
survive a redesign; pinning it to a selector throws that away. |
| Q |
How long should a test be? |
| A |
One test, one question. If a red result would leave you unsure which part failed, it is too long.
See Choosing What to Automate First. |
| Q |
The agent asked me a question instead of building. Is something wrong? |
| A |
No — that is the product working. It flags contradictions rather than building something
destined to fail. Read it; it is usually right. |
| Q |
Can I describe a flow with a screenshot instead of words? |
| A |
Yes. The + button on the prompt box attaches files, and for a fiddly screen a
picture is often clearer than a paragraph. See Starting a Session: The Studio Home Screen. |
Running tests and reading results
| Q |
Where do tests actually run? |
| A |
On a clean runtime host in our cloud, in a real browser. You can close the tab; the run continues. |
| Q |
Should I put my suite on a schedule straight away? |
| A |
No. Decide who reads the result and what they do when it is red, first. A scheduled suite nobody
reads manufactures confidence. |
| Q |
What is the difference between a test, an orchestration and a workflow? |
| A |
A test proves something and passes or fails. An orchestration is a group of tests run as one job. A
workflow is a scheduled job written in plain language that produces a result rather than a verdict.
See Workflows: Scheduled Jobs Written in Plain Language. |
When tests fail
| Q |
A test went red. Where do I start? |
| A |
Which step is red, and did the ones before it pass? Then read the Execution Error — it names
the comparison, the expected value and the value found. See Diagnosing a Failed Test in Functionize Studio. |
| Q |
How do I tell a broken test from a broken application? |
| A |
The Data tab. If Current differs from Previous Successful Run,
your application changed. If they match and it still failed, the expectation moved. |
| Q |
I clicked Fix Test and it refused to fix it. Is that a bug? |
| A |
No, and it is important. Studio repairs drift — a test looking in the wrong place. It
will not silently rewrite an assertion to match a page that genuinely disagrees with it, because
that would delete the failure you wanted to catch. It stops and asks instead. |
| Q |
My tests fail intermittently. |
| A |
Usually timing or environment rather than the product. Raise the Missing element
timeout, or move the Timing model toward Conservative. See
Self-Healing and Timing: Controlling How Studio Recovers. Do not tolerate flake — it
is how suites stop being read. |
| Q |
What should I include when I raise a ticket? |
| A |
The test ID, the browser and version from the run header, and the time of the run. Those three let
us find the exact execution instead of asking you to reproduce it. |
Test data and secrets
| Q |
How do I run the same tests against different environments? |
| A |
Execution presets. Define a variable set per environment and choose one at run time, rather than
duplicating the suite. |
| Q |
Can a later step use a value an earlier step captured? |
| A |
Yes — an order number or reference. Check it resolved in the Variables tab. |
Credits and plans
| Q |
What uses credits? |
| A |
Agent work: creating, running, and diagnosing or fixing tests. Manual edits, dashboards, reports,
exports, integrations and admin are free. The rule: if an agent is doing the work it uses credits;
if you are, it does not. See How Credits Work. |
| Q |
Why did our trial burn credits so fast? |
| A |
Because building is the heavy part and a trial is mostly building. One new test costs roughly what
fifty runs cost. Consumption is front-loaded and settles once your suite exists. |
| Q |
What happens if we run out? |
| A |
You are never locked out. Your tests, results, data, manual editing, dashboards, exports and admin
keep working. Only new agent work pauses until credits are added, you upgrade, or the next cycle
begins. |
| Q |
Do unused credits roll over? |
| A |
On self-serve monthly plans they reset each cycle. On Enterprise they roll over for the term. |
| Q |
How do we reduce consumption? |
| A |
Fix rather than rebuild, be specific up front so there are fewer regenerations, make small edits by
hand (free), reuse flows as components, and prune stale tests. |
| Q |
Will costs rise as we scale? |
| A |
Functionize runs its own purpose-built models, so costs are not tied to third-party token pricing.
Users are unlimited on Enterprise, so adding people does not add cost. |
Skills and integrations
| Q |
What is a skill and do I need one? |
| A |
A short document teaching the agent about your application. You do not need one to start, and it is
the biggest single lever on quality once you are going. See Why Skills Matter. |
| Q |
We already have a Selenium or Playwright suite. |
| A |
Use the migration skills to bring it across in reviewable batches — and treat it as a chance
to delete the tests that were never earning their keep. |
Strategy and planning
| Q |
Do we need a written test strategy, or can we just start building tests? |
| A |
Start building — one passing test teaches you more than a week of planning. But write the
strategy before you have forty tests, because that is the point at which nobody can remember why
any particular test exists. A strategy that fits on a few pages is enough. |
| Q |
Which template should I fill in first? |
| A |
The Master Test Strategy Template, and only sections 2, 4 and 5 — what you test
today, your critical journeys, and what you automate. That is an afternoon, and it is most of the
value. The rest can follow. |
| Q |
What is the single most important section in any of them? |
| A |
Section 2 of the master template: an honest baseline of what you test manually today, with the
date on it. Six months later it is the only evidence you have of where you started, and it is the
section teams are most tempted to skip. |
| Q |
How do I decide what to automate? |
| A |
Score each candidate on six questions: how often it runs, whether the outcome is deterministic,
what it costs if it breaks silently, whether the behavior is stable, whether you control the data,
and whether it runs independently. See What Makes a Good Automation Candidate, which has
the scoring bands. |
| Q |
How many critical journeys should we have? |
| A |
Usually ten to twenty. If your list has sixty, you have listed features rather than journeys. The
test is: if this broke at 2am on a Friday and nobody noticed until Monday, what would it have
cost? |
| Q |
Should we automate everything? |
| A |
No, and saying so explicitly is a mark of a mature program. Exploratory testing, subjective
judgment, anything needing a physical action, one-time migration checks, and behavior still being
designed all stay manual on purpose. |
| Q |
We are migrating from SAP ECC to S/4HANA. Do you have anything for that? |
| A |
Yes — a separate set covering TUT, FUT, SIT and UAT, written for how those programs are
actually organized and for the systems-integrator split. Start at Testing an SAP S/4HANA Transformation. It is also honest about which SAP screens a browser can drive: Fiori and SAP
GUI for HTML, but not SAP GUI for Windows. |
| Q |
How do we roll this out across several teams? |
| A |
Sequence them rather than launching them all at once, and agree five conventions everyone follows:
project boundaries, test naming, tier tags, the promotion rule, and who reads red. See
Scaling From One Team to Many. |
Best practices
| Q |
What is the biggest single thing we could get right? |
| A |
Never let the same AI session that wrote the code also write and judge the tests for it. Models
rate their own output more highly than equivalent output from anyone else, and a test written from
the implementation inherits the implementation's misunderstandings exactly. See
The Independent Tester Principle. |
| Q |
What makes a prompt produce a test that lasts? |
| A |
Describe what a user does and what must then be true — never how the page is built. Name the
starting URL, state the specific value that must hold, and add what must not happen. See
Writing Prompts That Produce Durable Tests. |
| Q |
Do we need to review tests the agent generates? |
| A |
Yes, and it is about ninety seconds each. The one question that catches most weak tests: if the
thing this test protects were broken, would this test go red? See
Reviewing AI-Generated Tests. |
| Q |
How should we organize the suite? |
| A |
Three tiers — smoke, critical, regression — scheduled differently, with a named reader
for each. Nothing enters a tier that gates a release until it has passed three times in a row. See
Keeping a Suite Green. |
| Q |
A test fails intermittently. Can we just rerun it? |
| A |
No. Flake is a defect, not weather, and there are exactly three options: fix the cause, demote it
out of any tier that gates something with a date, or delete it. Quarantine with no date is
how flake becomes permanent. |
| Q |
Who should own a red result? |
| A |
A named person per scheduled suite, with a timescale and a first action. This single decision
predicts whether the suite is still being read in six months better than anything else. |
Measuring quality in an AI-DLC
| Q |
What is an AI-DLC, and is it a standard? |
| A |
The AI-driven development lifecycle: an AI agent drives the work across the lifecycle while people
validate and decide, rather than authoring everything themselves. It is a useful frame and it is
not a standard — it is a vendor coinage from 2025, its own sources describe either three
phases or five, and analysts use a different label for the same shift
(AI-native software engineering). Adopt it deliberately rather than comply with it. See
AI Testing and AI-DLC: A Glossary. |
| Q |
Why does measurement matter more in an AI-DLC than it did before? |
| A |
Because the thing that got faster and the thing that got riskier are not the same thing. Three
independent results say so. DORA found AI adoption has a positive relationship with throughput and
a negative one with delivery stability. A randomized trial found experienced developers
took 19% longer with AI tools while believing they had been 24% faster. Analysis of 211 million
changed lines found copy-pasted code rising and refactoring falling. If you measure only speed, none
of that is visible. |
| Q |
Which of the five metrics moves most once agents are generating the code? |
| A |
Maintenance share and escaped defects. Writing tests is cheap now,
so test count stops carrying information almost immediately. What tells you whether the program is
compounding is whether the share of effort spent repairing tests is falling, and whether fewer
defects reach production. Both need a dated baseline to mean anything — the definitions and
targets are in What to Measure: QE Metrics and KPIs with Studio, which also has a
printable tear sheet. |
| Q |
We already track DORA metrics. Does anything change? |
| A |
Two things. Add a stability measure if you only track throughput, because that is precisely where
the AI-adoption penalty shows up. And check your definitions: DORA has retired MTTR, which expanded
three different ways in three different standards. The current pair is failed deployment
recovery time and a rework rate. |
| Q |
Self-reported productivity gains — are those worth reporting? |
| A |
Not on their own. The METR trial is the cautionary case: the developers who were 19% slower
believed they were 24% faster, and they were not being careless. Feeling faster and being faster
came apart. Pair any survey number with a measured one. |
| Q |
Can we measure the AI itself, or only the code it produces? |
| A |
Both, with different instruments. Code an agent wrote is tested the ordinary way — it is just
code, and it needs the same independent checks. If your own product contains a model, you
need evals, which is a different discipline: a test case is a distribution rather than a pass or a
fail, and pass rate has to become a rate across repeated runs. See Evaluating and Testing AI: Terms. |
| Q |
Our coding agent runs the tests and reports green. Is that enough? |
| A |
No, and this is the sharpest risk in the whole model. An agent that runs your suite treats green as
its stopping condition, so a weak test now actively licenses bad code rather than merely failing to
catch it. Keep the thing that verifies independent of the thing that builds — see The Independent Tester Principle. |
| Q |
A vendor says they cut test maintenance by 70%. How do we check that? |
| A |
Ask for the definition, the denominator and the baseline, in writing. There is no standard
definition of maintenance share, and ISTQB does not define "flaky", "self-healing", "autonomous
testing" or "code coverage" either — four of the words such claims lean on hardest. The
glossary flags 56 contested terms for exactly this conversation. |
| Q |
Where do we start if we have measured nothing so far? |
| A |
Half an hour, this week, before building anything: escaped defects last quarter, manual regression
hours per release, and which critical journeys are covered today — with the date written on
it. Section 2 of the Master Test Strategy Template is that page, already laid out. It is
the section teams skip and the only evidence you will later have of where you started. |
Team, scale and proving value
| Q |
What should we measure? |
| A |
Escaped defects, critical journey coverage, time to feedback, maintenance share and flake rate.
Capture a baseline in week one or you will improve things and be unable to prove it. See What to Measure: QE Metrics and KPIs with Studio. |
| Q |
How do we justify the spend month over month? |
| A |
Hours displaced, escaped defects prevented, and credit consumption with its front-loaded shape
explained. Month one looks build-heavy; show the trend rather than the snapshot. |
| Q |
How many tests should we have? |
| A |
Wrong question. Ask instead: if every test passes, what do we now know is true? If that does not
include "customers can do the thing we charge them for", the suite is not finished regardless of
size. |
| Q |
When should we bring the rest of the team in? |
| A |
Around day six, once the environment works — so the session teaches rather than debugs. Give
everyone the exercise series and its printable lab handouts. |
| Q |
Something is not covered here. |
| A |
Start at Start Here, or raise a ticket. If you are stuck
on access, a skill, or a migration, ask in week one rather than week six. |