In a release test I published in July, Katalon’s AI runner reported seven cases passed and four failed. I could open the steps, inspect their screenshots, and watch the browser session. The failed rows showed which expectations broke. The passed rows deserve the same scrutiny: what had the agent observed before it called each step passed?
An AI test runner has to perform the action and judge the outcome. A false pass is a passed verdict when the expected behavior did not occur. Run with AI in Katalon True Platform executes natural-language manual cases in a browser and records step results, an AI log, screenshots, and video where supported. Those records help a tester check the verdict. They do not make every verdict correct.
Consider a hypothetical checkout test: apply a valid 10% discount to a $100 subtotal, then submit the order. The agent enters the code, clicks Apply, sees “Saved,” and moves on. If the discount line is $0, that step failed even though the page accepted the click. If the discount line is $10 but the saved amount due is still $100 before tax, the checkout flow needs another check.
The test oracle is the source of the expected answer. For this example, the oracle is the pricing rule and the resulting order value, expressed in the test case. A toast is evidence that an interaction occurred. It cannot answer a question about the saved amount.
This is where my first dogfood cases struggled. I had packed setup, navigation, action, and judgment into large natural-language steps. The resulting verdicts were vague. I split the cases into one action and one checkable expected result per step, then used separate Playwright tests for timing and streaming checks that required code-level precision. The runner could report which expectation failed, and I had a place to inspect the evidence for that step.
For the checkout example, the step should say what to inspect: “Apply code SAVE10 to the $100 subtotal. The discount line shows $10.” A later step can check the saved order. The tester still has to choose the right tax rule, data, and source of truth. An agent cannot infer a business rule that the case never states.
Ask a courier to deliver a blue notebook. A photo of the courier at the right house proves the route worked. You also need to see the blue notebook at that house. Browser actions are the route. The expected result says what must be delivered. A pass needs evidence for the expected result.
The current Run with AI guide describes a human-in-the-loop manual run. In AI Sessions, the Test Steps tab shows step status; the AI Log records actions, generated scripts, results, and failure details. The Screenshots tab shows visual evidence. Video is available in supported environments, including Windows. When a step fails, AI Verdict gives an assessment and suggested next action. A pending review banner asks the tester to inspect AI-assisted results before ending the run.
Each record has a different job. The status is the agent’s conclusion. The log explains its route through the page. A screenshot or video shows what was visible at the time. A reviewer compares these with the expected result. For a false pass, the most revealing clue may be missing evidence: the agent reported success after seeing the toast, but no captured state shows the discount or saved order amount.
The environment can also prevent a valid check. Secrets & Variables provide project values for cloud-hosted runs, including Run with AI. A case can refer to a saved key instead of putting a password in its text; the documented flow stops when a required value cannot be resolved. Katalon Tunnel can connect a cloud run to a private web application while its client has network access. Today’s documented runner may ask a person for guidance during execution. A blocked login or unreachable application needs to appear as such in the review, rather than being mistaken for an application failure.
The following describes the runner that handles production runs as of September 30, 2026. A separate runner, built around an execution agent and an independent referee agent, is in development and is not enabled for any organization.
The runner starts with the authored step and the current browser state. Grounding means translating that instruction into an action against a real element on the live page. The code tries a fast structural match first, then model-assisted reading of the page’s DOM or visual state when needed. It checks a selected target against the live page before acting. This helps when a button moves or a label changes, but a correct click is only the first half of the test.
After the action, an observer checks the expected postcondition on the resulting page. It allows time for a transition and limits how often it repeats the model check as the page changes. If the action fails, a recovery path can choose another interaction and run the postcondition check again. The agent can adapt its route, while the stated expected result remains the thing it must prove.
Failure diagnosis has a narrower job. The code collects evidence, checks concrete runtime signals before asking a model to interpret ambiguous cases, and can return “insufficient evidence.” In its default summary mode, that advice does not rewrite the recorded test status. The distinction matters because a plausible explanation of a failure is not evidence that the test passed.
None of this supplies an oracle when the case lacks an expected result. “Open the order page” can be completed by reaching a page. “Open the order page and verify that the saved amount is $90 before tax” demands an observable assertion. The runner’s reasoning is useful for finding the control, coping with page changes, and checking a stated condition. Case authors still decide which condition represents quality.
An execution success rate counts whether the runner completed actions. It does not reveal how often the verdict was wrong. To evaluate an AI test runner on a real library, we need cases with independently known outcomes, then inspect both false passes and false failures. We also need to record cases the runner could not execute because of missing data, access, or an ambiguous step. A small demonstration run teaches a team how to read the evidence; it cannot establish reliability across the team’s library.
My July release workflow used two execution lanes: natural-language cases through Run with AI and code-based Playwright tests, with results visible in True Platform. The Playwright lane handled precise checks such as timing thresholds and streaming behavior. When a natural-language case reports a pass, I still need its expected result and the observation that supports it before that result can inform a release decision.