In one release test, my AI agent triaged 43 backlog items into 134 executable cases in Katalon True Platform. The first pass at the steps caused trouble. We had bundled setup, navigation, action, and judgment into long instructions. The runner returned verdicts, but a failed step left me unsure which expectation it had checked. We split the cases into one action and one checkable expected result per step. The failures became specific enough to review against the screenshots and session recording. I wrote up that run and the cases it produced.
The case count told me how much we had written. I still needed to know whether the application had done the right thing. Katalon AI is the testing assistance built into Katalon Studio and Katalon True Platform. It helps people use requirements to design cases, act through testing tools, and review what happened. Studio focuses on automation work in the IDE. Katalon True Platform works with requirements, test cases, runs, failures, and project data. The two assistants serve different tasks and use different context.
This is testing with generative AI, in ISTQB's terminology. It includes test design and execution, then the harder work of deciding whether a reported pass or failure is credible.
Take a ticket that says, “Returning customers can reorder.” A model can write plausible scenarios from that sentence. A tester still needs to know whether an unavailable item may be reordered, which price applies after a price change, what account state to use, and what the screen or stored order should show. If those answers are absent, fluent test steps can turn guesses into expected results.
We built the Katalon True Platform requirement workflow so a tester can examine the source requirement, answer clarification questions, add an application URL or test data, then review draft cases before saving them. When Jira or Azure DevOps requirements are synced into Plans, the saved cases can be linked back to the originating requirement. That link is useful only if the case's expected result follows the requirement the team meant to test. A reviewer still has to check it.
The AI knowledge base can give the assistant a project's cases, requirements, and uploaded documents. It is in limited availability and not yet on for most projects, so searching existing coverage remains a review task. Connected external apps can add context when configured. A source outside the product is not automatically present in every project.
Context engineering is a useful name for this work: selecting the information an agent needs for a particular decision. In testing, the difficult part is often deciding which version of a rule and which existing cases belong beside the ticket. The generation article goes further into testable requirements and reviewable cases.
Imagine asking a new teammate to check a reorder button. If all they have is “make sure reorder works,” they can click it and guess. Give them the current rule for unavailable items, a customer account, and the expected confirmation, and they can tell you what they checked. Katalon AI needs that same case file. It can help assemble and use the information. The team must still settle an unclear rule.
There is a practical difference between a workflow with preset steps and an agent that can choose its next action after reading a result. Anthropic describes that distinction. In Katalon True Platform, the assistant routes a request to agents and tools for work such as finding a requirement, generating cases, starting a run, or investigating a failed result. For the reorder ticket, reading the source before creating a case is a useful choice. If the unavailable-item rule is missing, the requirement-analysis step is where that question should come up, before anyone writes an expected result from a guess.
The Katalon True Platform MCP server makes many of those testing operations available to compatible external AI agents. Model Context Protocol, or MCP, gives an agent named tools for work with test artifacts and results. For example, find_requirements reads synced Jira or Azure DevOps requirements. create_test_case saves a manual case. Other tools work with runs and results. Approval cards and draft review belong to the embedded assistant. A write through MCP is saved immediately, with Draft status and no review stage, so an external agent needs its own review policy.
In my release dogfood run, a terminal agent used those platform operations to organize cases, start executions, collect results, and file product friction to the backlog. The workflow was written as a skill that told the agent when to classify a backlog item, how to name cases, and what evidence to return. That made the agent's choices reviewable without prescribing every click.
A search result and a new test run have different consequences. Katalon True Platform's tool approval documentation classifies every tool as a read or a write, and starting a run counts as a write. Where approval controls are enabled, the embedded assistant presents a card showing the proposed change and waits for the user to allow or deny it. Approval never grants access the user lacks. The controls are still being rolled out organization by organization, so this behavior should be checked in the workspace being described.
A user can choose Always allow for a particular tool in their own browser. Future writes through that tool then run without a card and remain recorded in the conversation. That choice does not apply to teammates or other browsers. It also makes the scope of a tool important: approving create_test_case is a broader decision than approving one case title.
Review continues after an action. A generated case can contain a wrong business rule even when the save operation succeeded. A run can complete every browser action while misreading the outcome. The artifact and the evidence need inspection at the point where a team decides to rely on them.
Testers call the basis for an expected result a test oracle. If the reorder rule specifies the current price, a case can check that the customer's new order shows that price. “Click through the reorder flow” describes activity, but gives the runner little basis for a pass or fail.
Run with AI operates a browser from natural-language manual cases. The AI Sessions view shows step status, the AI log, screenshots, and video where supported. A reviewer can compare the stated expectation with the recorded browser state. Windows AI runs record step results, screenshots, and video. The current product also asks for human guidance during a run when the assistant needs it.
The release test gave me a second way to inspect some behavior. I used Playwright for checks that needed coded assertions, such as a timing threshold, and sent those results into the same Katalon True Platform project. The published write-up shows the AI-run cases and Playwright results side by side. If their verdicts disagree, I have a concrete investigation: compare the case wording, test data, environment, browser evidence, and code assertion. The execution article examines false passes and the evidence behind a verdict.
The 134 cases were the platform case count. Playwright checked selected behaviors separately. Each result gave me a case, an expectation, and evidence I could open.
To see requirement clarification, Run with AI and the evidence behind each verdict in your own project, explore Katalon TestOps, part of Katalon True Platform.