.svg)
Claude Code shipped workflows: reusable, versioned instructions an agent can follow instead of improvising a process from scratch each time. We used that feature to hand Claude Code our entire QA job.
Not "write a test case." The whole loop: read the requirement, design the coverage, run it on the platform, file the defect, report back with evidence. The kind of work a QE does in a day, compressed into one skill triggered with a sentence. It's one concrete answer to what agentic QA actually looks like when someone builds it instead of just describing it.
Here's how it works.
THE SHORT ANSWER
A Playwright AI agent, built as a Claude Code skill on Katalon True Platform, at a glance:
• One skill, three lanes: manual + Run with AI, automated/Test Execution Cloud, or Playwright, routed by where the work already lives.
• Playwright is a first-class lane: real .spec.ts files, run in CI, with results uploaded straight into True Platform via @katalon/playwright-reporter.
• Playwright MCP (Microsoft's) and Katalon's MCP server solve different problems: one drives a browser, the other manages test artifacts. This agent uses both, for different jobs.
• The agent reads a requirement, designs risk-based ISTQB coverage, executes it, and reports back with evidence, not "looks broken."
The rest of this walks through the lane router, the Playwright lane in depth, and how to get it running on your own project.
Two Different "Playwright + MCP" Things, and Why That Matters
Before any of this makes sense, one confusion needs clearing up, because it trips up almost everyone the first time they hear "Playwright" and "MCP" in the same sentence.
Playwright MCP is Microsoft's own open-source MCP server (@playwright/mcp). It lets an AI agent drive a real browser: navigate to a page, read the accessibility tree instead of guessing from a screenshot, click, fill fields, and generate test code from what it just did. It's what lets Claude Code, Cursor, or Codex CLI touch a live web page at all. Microsoft's own docs and GitHub repo are the reference for the full tool list.
Katalon's MCP server, plus @katalon/playwright-reporter, solves a different problem: it lets an agent manage test artifacts on a platform, create test cases, link them to requirements, kick off runs, read results, and pull already-executed Playwright results into a shared dashboard. It has no opinion about how the browser gets driven. It just needs the outcome.
The skill we built uses both, for different jobs, and that split is the golden rule underneath the whole thing: use a browser or Playwright MCP for anything that touches the live application under test. Use Katalon MCP tools for anything on the platform, projects, requirements, cases, suites, runs, results, defects. Never make an agent scrape the platform with a browser when an MCP tool already exists for it, and never make it guess at page structure when it could read the accessibility tree instead.
It's also the concrete version of a claim we make about our own architecture: that no competitor spans dev tools and testing data in one shared context layer the way an MCP-connected agent can. An agent that calls both a browser and a test management platform from the same conversation is what that claim looks like in practice, not on a slide.
The Setup: A QE's Real Job Is Decisions, Not Keystrokes
A QE's real job isn't writing test scripts. It's a sequence of decisions:
- What changed, and what's the risk?
- What's the smallest set of tests that covers that risk?
- Which way should it actually run?
- Did it pass, and can that be proven?
Every one of those is judgment. None of them is "type the Selenium." So the real question isn't whether AI can write a test, it's whether it can hold the decisions and drive the platform.
It turns out it can, if the workflow gets handed over instead of the keystrokes.
The Trick: Automate the Decision Tree, Not the Clicks
The old way to "automate QA" was to record clicks. Brittle, and it skips the only part that matters: the thinking.
The better way: write the decision tree down once, hand it to the agent, and let it route every task through it.
The whole skill is built on one decision: which lane runs this test.
| Which Lane Runs This Test? One branch routes every task; the lane is chosen by where the work already lives. |
|
|---|---|
| LANE A Manual + Run with AI |
New behavior, no code yet, exploratory. Design cases → AI executes them against the live site. |
| LANE B Automated / Test Execution Cloud |
Automation already exists in the repo. Schedule the suite on the grid. |
| LANE C Playwright |
Team lives in code, wants CI. Real .spec.ts files, results upload automatically. |
That one branch is the whole brain. Everything else, requirement analysis, ISTQB coverage, suite building, reporting, hangs off it.
What It Actually Does, Start to Finish
Take a prompt like: "We shipped a new product filter on the storefront. Cover it."
The agent:
Reads the intent. Pulls the requirement from Jira or Azure DevOps, or analyzes the one-liner if there isn't one. Produces personas, main flow, alternate flows, negative flows, and risk areas.
Designs the coverage, risk-based, not maximal. It applies real ISTQB techniques: equivalence partitions for filter values, boundary analysis for the price slider, a decision table for filter plus stock plus sort, state transitions for the result list. Then it writes a coverage note: what it's testing, what it's deliberately skipping, and why. The "why" is the part junior testers skip and senior testers rely on.
Routes to a lane. New behavior with no automation yet routes to Lane A. It drafts human-readable cases, imports them into Katalon True Platform, links them to the requirement for traceability, builds a suite, picks the AUT environment, creates a manual run, and starts Run with AI, the platform's own agent that executes the cases against the live site.
Reports with evidence. Pass, fail, and blocked counts. Each failure comes with the verbatim error, a screenshot, and a trace, not "looks broken." If it can't produce the proof, it downgrades the claim. Then it offers to file the defect against the failed result.
One sentence in. A traceable, executed, evidenced run out.
Playwright as a First-Class Lane
Most "AI QA" demos stop at the manual lane. But plenty of teams live in code: they want Playwright, cross-browser coverage, and CI gating merges before anything ships.
The integration behind Lane C is @katalon/playwright-reporter: existing @playwright/test specs run anywhere, and the results, status, duration, screenshots, videos, traces, browser metadata, upload straight into True Platform's Test Runs. The dedicated Playwright integration guide already covers install, config, and CI upload options in depth, so this section focuses on what the agent does with the results once they land.
The strongest idea in the whole workflow is the promotion pattern: explore the live site with a browser, or Playwright MCP, to cover new behavior fast in Lane A. Once the happy paths are stable, port them to Lane C Playwright specs so CI gates every future merge.
One product. Three lanes. One place to read the results. The agent knows when to use which, and flags when it's recommending a promotion versus a one-off.
What Stood Out as We Built It
Honesty beats capability. The most useful part of the skill wasn't a feature, it was the boundary table: "You cannot create requirements in Katalon, they sync from Jira. Here's the workaround." "You cannot guarantee Run with AI finishes, report the blocked state and the exact fixture that's missing." An agent that knows what it can't do is worth ten that bluff.
The decision tree is the product. The value isn't in writing more tests. It's in building the harness that lets an agent decide which tests matter, run them, and know when the result can be trusted.
Fewer, stronger tests win. Left alone, an agent will happily generate 200 shallow cases. The ISTQB guardrails, prefer fewer strong tests, mark P0 for revenue, checkout, and data-loss paths, are what make the output something a QE would actually sign off on.
Why Workflows Change This
Before workflows, this was a prompt re-typed and re-tuned every time. Now it's a skill: written once, versioned, triggered by intent. The agent doesn't improvise the process, it follows it, and improvises only the judgment inside each step. That's the difference between a clever demo and something a team would let near a release.
We gave Claude Code our QA job. It didn't take it. It gave us back the eight hours we spent on the mechanical 80 percent, and handed us the 20 percent that's actually judgment. That's the trade we'll take every time.
Get It Running
Two moving parts, both public.
1. Connect the Katalon True Platform MCP server. This gives the agent the platform tools: creating test cases, reading AUTs, opening manual AI sessions, scheduling test runs, and the rest.
The first tool call triggers a browser sign-in, then asks which workspace to use.
2. Install the harness. Katalon publishes an open-source skills repo, katalon-labs/true-skills, that installs the lane router, the ISTQB coverage guide, and the capability-boundary table as a Claude Code plugin, along with a matching skill for the Playwright lane specifically.
It mirrors the lane-routing pattern we built by hand before the official skill existed: connect the MCP, add the reporter for the Playwright lane if needed, install the harness, and type a plain-language testing task. The router does the rest.
The repo's katalon-trueplatform-testing skill is the direct successor to the one-command auto-test wrapper this workflow started from: same requirement-to-result loop, triggered the same way, with a prompt like "Test requirement CEL-6 end to end on Katalon Platform." For readers who want to see the raw lane-router prompt this was built from, or the original downloadable bundle, the original walkthrough has both.
It's built against Katalon True Platform, a public product, and the public @katalon/playwright-reporter package, nothing vendor-internal. Take it, change it, ship it.
Huy Tieu is Product Lead at Katalon, building agentic QA. This piece adapts his original post for the Katalon blog. For more on where he thinks agentic QA is headed, see his Quality People interview, and for a look at the six AI agents built directly into True Platform, see how those agents actually work.
|
