Every testing vendor says "AI-powered" now. Fewer of them can show you the actual verification pipeline underneath that claim: what gets checked, what gets trusted without checking, and what happens the moment something breaks. This is that pipeline, as I run it against True Production Insights, formerly known as TrueTest, every time we ship a change to it.
True Production Insights is the production insights capability of Katalon True Platform. It maps real user journeys against existing test coverage and auto-generates the tests that are missing. Testing a feature like that means testing something that itself generates tests, which raises an obvious question: how do you verify an AI system whose job is to write and run other tests, without just trusting its output on faith?
THE SHORT ANSWER
How we smoke test True Production Insights with AI:
• An orchestration layer, built on OpenAI Codex, coordinates the pipeline; Playwright and Katalon Runtime Engine still do the actual browser automation and execution
• Every transformation in the chain, from a browser interaction to a generated test script, is verified independently, not assumed correct because the next step happened
• Test contracts stay human-authored and reviewed in Katalon True Platform, AI does not get to redefine what "passing" means
• When something fails, an AI diagnostic assistant preserves evidence and classifies the failure before any retry, and escalates anything uncertain to a person
• Generated tests get executed, not just produced, so "the AI wrote a test" and "the test actually works" are never treated as the same claim
The rest of this post walks through why we built it this way.
The value of True Production Insights depends on a chain of handoffs, not one interface: capturing real user traffic, identifying sessions, mapping journeys, generating test cases, then executing them. A bug in any single handoff can produce a plausible-looking result that's actually wrong. So instead of testing "does the UI show the right thing," we test whether each handoff in that chain does what it's supposed to.
The way we've set this up, an orchestration layer sits above the deterministic tools rather than replacing them. OpenAI Codex coordinates execution and decides what runs next. Playwright handles the actual browser automation and evidence collection, screenshots, videos, network activity. Katalon Runtime Engine executes the generated test cases. None of those tools changed what they do. What changed is that something now coordinates them and checks their work.
The seven-stage True Production Insights engine pipeline, from AUT behavior to runtime execution, with Codex orchestrating and Katalon True Platform recording the result.
Before any of this runs, someone has to define what "correct" means. That definition, the test contract, is human-authored and stored in Katalon True Platform, not inferred by the AI mid-run. This matters more than it sounds like it should: an AI system that can quietly redefine what a passing test means is a system you can no longer trust the pass rate of. Contracts stay fixed. The AI's job is to execute against them and report honestly when something doesn't match, not to decide the goalposts moved.
The reviewed suite behind every smoke run: six human-authored test cases, each one the actual contract the AI executes against.
We don't point this pipeline at live production traffic and hope for clarity. We generate deliberate synthetic journeys instead, product browsing, searching, filtering, cart operations, checkout, in a sample ecommerce application, and tag every browser context with a unique smoke-run identifier. That single decision is what makes the whole pipeline debuggable: when something goes wrong, we know exactly which run, which session, and which step produced it.
Verification happens before downstream processing gets a chance to run, specifically so a tracking failure shows up as a tracking failure. Waiting for downstream processing would only turn a clear collection failure into a confusing "sessions not found" failure later, by the time you're debugging the symptom instead of the cause, you've lost the context that would have made it a five-minute fix.
The full chain looks like this: a Playwright interaction becomes a tracking request, which becomes a captured user-session event, which becomes a journey-map page and action, which becomes a generated test object and script, which becomes a runtime result. We check each link independently rather than assuming a downstream success proves every upstream step worked correctly. A test can pass for the wrong reason. Checking only the final result would let that slide through undetected; checking every transformation is what catches it.
When a test fails, the easy move is to retry it and hope it passes the second time. We don't do that. An internal AI diagnostic assistant, we call it Thor, preserves the evidence first: traces, screenshots, video, network data, console output. Only then does it classify what actually happened, an automation defect, a timing issue, a genuine product defect, or an environment problem, since the fix for a flaky selector is nothing like the fix for a real regression. If the classification comes back uncertain, it escalates to a human rather than guessing. Automation issues get corrected with a focused retest. Nothing gets silently re-run into a pass.
This is the principle the whole pipeline exists to enforce. An AI system that generates a test case has not yet demonstrated anything, it's produced an artifact, not a verified fact. So every generated test gets executed, every time, before it's treated as trustworthy. "The AI wrote a test" and "the test works and means something" are two different claims, and we don't let the first one stand in for the second.
Every smoke run produces a conclusion-first HTML report: overall pass/fail status up top, results mapped back to the human-authored test cases they correspond to, and a direct link to diagnostic evidence for anything that failed. Step-level outcomes publish to Katalon True Platform, and we preserve first failures for historical comparison rather than overwriting them on the next run. A recent run we reviewed covered six test areas: six cases reviewed, four passes, two failures, each with a clear evidence trail rather than a bare red X.
The Tests tab of an actual production smoke run: four passes, two failures, each traceable back to its own test case and evidence.
The work shifts from writing and babysitting scripts to reviewing evidence and making the call on disposition. That's not a smaller job, it's a different one: less time spent on script maintenance, more time spent deciding whether a flagged issue is real. It's the same "editor, not author" shift we've built into the rest of the AI agents across the platform, just applied to our own internal QA process instead of a customer-facing workflow.
Nothing about this reasoning pattern, test intent plus execution adapter plus observable evidence plus an independent runtime check plus human-governed disposition, depends on Playwright specifically. Selenium, Cypress, WebdriverIO, or Appium could sit in the execution-adapter slot without redesigning anything upstream of it.
The orchestration layer coordinates whichever framework is doing the driving; it doesn't care which one that is. That's what makes this a durable approach rather than a Playwright-specific trick, and it's the same principle behind Playwright best practices we follow elsewhere: build around stable contracts and observable evidence, not around any one tool's internals.
A few things held up across every run we've done. Deterministic assertions work better for measurable facts than fuzzy AI judgment calls do. Giving every synthetic batch a unique identity is what makes debugging tractable later. Failing at the earliest broken layer saves hours compared to chasing a symptom three steps downstream. And preserving original evidence before retrying anything is non-negotiable, once you retry first, you've usually destroyed the thing you needed to diagnose.
None of this makes the case that AI should replace frameworks or people. It makes the opposite case: AI testing earns trust by expanding what gets checked and keeping every check explainable, not by removing the humans who decide what "correct" means. That's not a slogan for us, it's the actual shape of the pipeline that ships changes to True Production Insights every release. If you're evaluating whether an "AI-powered" testing claim holds up, this is the kind of verification architecture worth asking to see.