Keep testing healthy

Qarmiq tells you how stable your product is. How far you can trust that reading depends on the habits behind it. A radar fed by stale tests, expired access and an ignored backlog looks calm for the wrong reasons.

This guide is the operating model: eleven habits, who should own each, the cadence, and the numbers on Insights that tell you whether it is working. The cost shows up when it is not done.

What it costs when it slips

If you let this slide	The consequence
Decisions pile up	Suggested scenarios never become coverage, and findings and changed pages go unacted on.
Environments are not crawled	Qarmiq’s picture of your product ages, and coverage stops being real.
Sign-in access expires	The agent cannot reach the product, and tests quietly stop covering what matters.
Reports go unread	Real failures and flaky results sit in plain sight.
Nothing is run on a rhythm	Scenarios go stale, and a green radar means little.

Who owns what

Habit	Suggested owner
Decisions, reports, flaky scenarios, defects	QA or test lead
Crawls, sign-in access, scripts	Engineering lead or platform
Red lines, collections, scenario list	Product and engineering leadership
Access and keys	Whoever administers the project

Pick one named person per habit. Shared ownership is how these lapse.

The habits

1. Keep Decisions close to empty

Everything Qarmiq proposes waits on Decisions until a person acts. A growing list means nobody is steering.

Accept what is right, Reject what is not, Assign what needs someone else. A rejected suggestion is not made again.
Watch Open decisions on Insights. In HEALTH SIGNALS, the Decisions card shows how many have waited over 7 days and the median time to act.

2. Crawl your environments regularly

Crawl reads the pages of an environment and keeps Qarmiq’s picture of the product current.

Crawl after each release that changes pages, and at least monthly.
A crawl raises a Route change decision when a form field is removed or changed, so affected checks get updated.
Answer the questions on Knowledge. They sharpen future scenarios and tests.

The AI agent can only test what it can reach.

Add a role for each kind of user you need to test as, using accounts made for testing.
Save a username and password per role so Qarmiq signs in again when a session ends.
Act quickly on Session expired, Sign-in failed and Sign-in needed. Pause roles you are not using and remove ones you no longer need.

4. Read the reports

After each run, open the run report and start with WHAT NEEDS ATTENTION.
Read HOW STEADY THE RESULTS WERE. A scenario that changes its answer between runs is not yet trustworthy.
Read ALSO NOTICED BY THE AGENT. Unexpected problems show up there first.
Review Insights weekly. Ask Zoe about anything you do not follow.

5. Mark and watch red lines

Mark the few scenarios that must never fail as red lines. A breach stays open until the scenario holds again, and the breach button at the top of every page lists them. See Tags, entry points and collections.

6. Run on a rhythm

Today a person starts every run, so the cadence is a team habit.

Run the Smoke collection on every build and Regression before releases.
Aim to test every scenario within 30 days. Test freshness measures this, and a scenario untested for 60 days is Stale.
Watch SUITE CADENCE for a collection that has gone quiet.

7. Fix the cause of flakiness

A scenario marked Flaky gives different answers on repeat runs in the same browser and environment. Re-running until it passes hides the problem. Find what varies, such as data, timing or shared state, and fix it. Low Consistency pulls the stability verdict down.

8. Keep defects current

Move defects through their statuses as work happens, and close what is fixed.
Open critical and major defects lower the Defect-free signal until closed. See Defects.
File from a result or a finding so the evidence comes along.

9. Keep scenarios and checks current

Update wording when behaviour changes on purpose. A changed since chip on a run shows where a scenario moved on.
Merge duplicates, remove checks that no longer matter and add a check when a defect slips through.

10. Turn passes into scripts

When the agent passes a scenario, build a Playwright script. Scripts run first and cost no AI call. Look at any Not verified script and let it update.

11. Look after access

Add people as they join and remove them when they leave. Revoke API keys that are no longer used. See Project members and Profile and API keys.

The rhythm

When	Who	Do this
After every run	QA lead	Read the run report. File defects. Clear what the run raised on Decisions.
Every day or two	QA lead	Work Decisions to empty or near it.
Weekly	QA lead with engineering lead	Read Insights and the stabilisation plan. Check red lines, open defects and that sign-in roles are connected.
After each release	Engineering lead	Crawl, answer new questions, update scenarios and checks that changed.
Monthly	Leadership	Review collections, tags and red lines. Merge duplicates. Revoke unused keys and remove people who left.

A short weekly review of Insights, led by the QA lead, is a good way to keep the whole system honest.

Signals that it is slipping

You see on Insights	It usually means	Do
Open decisions rising, many waiting over 7 days	Nobody owns the list.	Name an owner and clear it.
Test freshness falling	Runs are not happening on a rhythm.	Schedule the collections.
Route coverage low or oddly flat	Entry points do not match your routes, or crawls are stale.	Fix the paths and crawl.
Result consistency low	Flaky scenarios, or too few repeat runs.	Fix the flakiness and rerun.
A collection tile stuck at 0 clean runs in a row	The release gate is failing or not run.	Look at the last run and decide.
Radar pulled in on Defect-free	Critical or major defects are open.	Triage and close them.
Verdict reads Not yet assessable	No recent verdicts at all.	Run something.

How you will know it is working

Open decisions near zero, with few waiting over 7 days.
Test freshness and Route coverage high.
Result consistency rising and nothing Flaky left unexamined.
Collection tiles above zero clean runs in a row.
A radar that fills out, and a verdict that moves toward Stable on strong evidence. See How to interpret the radar.

Questions to ask your team

Who owns each of these habits, by name?
How many decisions have been waiting more than a week, and why?
When did we last crawl, and are our test accounts still signed in?
Which scenarios have we not run in a month?