A conversation with Huy Tieu, Senior Product Manager at Katalon, on why the scripted testing model broke, what replaces it, and the one experiment every QA leader should run this week.
The term "Agentic QA" is everywhere in 2026. Gartner predicts 40% of enterprise applications will feature task-specific AI agents by year's end, up from under 5% in 2025. Forrester renamed its entire testing category from "Continuous Automation Testing Platforms" to "Autonomous Testing Platforms" last year. The industry has clearly decided that something fundamental is shifting.
But strip away the analyst reports and vendor positioning, and a simpler question remains: what actually changes when you hand testing work to an AI agent, and what happens to the humans who used to do it?
We sat down with Huy Tieu, Senior Product Manager at Katalon, to find out. Huy's path is unusual. He started his career as a Test Engineer over a decade ago, moved through business analysis and project leadership, and eventually crossed into product management. That QA-to-product pipeline gives him a perspective most product leaders don't have. He's spent the last year leading the development of Scout, an AI quality companion built on AWS's agentic AI technologies. And he has strong opinions about where this is all heading.
There's a definition problem in this space. "Agentic QA" gets thrown around to describe everything from self-healing test locators to fully autonomous testing agents. Huy draws the line clearly.
"Agentic QA is where the agent holds the product's intent, not a test script, and decides what to verify against it," he says. "The QE's job moves from writing steps to defining what 'quality' even means for this product."
He points to a concept that OpenAI formalised earlier this year: harness engineering. The idea is that the primary job of a team is no longer to produce the output directly. It's to build the system that lets agents do the work reliably. In QA terms, you stop writing test cases and start building the harness that lets an agent decide which tests matter, run them, and know when the result should be trusted.
"And to be clear about what it is not," Huy adds. "It's not self-healing locators, or AI-generated test scripts, or a chatbot that writes Selenium for you. Those are old-process-with-AI-sprinkles. If the shape of your QA workflow didn't change, if your team still writes and maintains scripts at the end of the day, you have AI-assisted testing, not agentic QA. Agentic QA is a different process. The old one doesn't survive the transition."
That distinction matters. The difference between AI-assisted testing and truly agentic approaches is not one of degree. It's a fundamentally different operating model.
If scripted automation worked well enough, nobody would be having this conversation. Huy's explanation for why it didn't is blunt.
"Scripts are a snapshot of a product that no longer exists. By the time the test is stable, the feature has shipped three versions. Your automation suite is always chasing a reality that's already moved on."
The deeper issue is compounding debt. Every new release adds more things to test, but teams still have to maintain the scripts from old features. After a few release cycles, the team spends more time keeping the automation green than actually testing the product. It's a treadmill.
This is the root cause of what Forrester identified as the industry's 25% automation coverage ceiling. Teams hit a point where the cost of maintaining the suite exceeds the value it provides, and they stop adding coverage. Not because they want to. Because the economics force them to.
"Agentic changes the math in two ways," Huy explains. "First, when the agent works from intent rather than instructions, you're not maintaining a thousand brittle scripts. You're maintaining a much smaller set of definitions about what 'good' and 'broken' look like. Maintenance stops scaling linearly with features, because the agent re-interprets intent each time the product changes."
"Second, agentic testing offloads the repetitive work. The 80% of regression that your QEs don't want to do, because it's mind-numbing and mostly green anyway, is exactly what the agent is good at. That frees the human team to focus on the parts that actually require judgment. The ceiling breaks not because agents are smarter than testers. It breaks because they're tireless, and the humans go spend their time on the work that matters."
The obvious pushback against handing test decisions to an AI agent is trust. If the agent decides what to test and how to test it, how do teams build confidence that it isn't just confidently missing things?
Huy doesn't dodge this. He leads with a failure.
"When we were building an early version of our AI Test Runner, the agent was too eager to 'complete the mission.' Its goal was: run the test case to completion, successfully. And it turned out the agent would work around the system. It tolerated weirdness, retried failing steps, took creative paths, all just to finish every step green. It was doing everything except testing."
"A real QE's job is to poke the product until it breaks. The agent was trying to get through the script like a checklist. Every step passed. The product could be broken and the agent would still call it a win."
The fix wasn't a smarter model. It was the harness.
"We rewrote the boundary: your job is not to complete the steps; your job is to find where the product breaks intent. We told the agent to be skeptical. If a step completes but the app isn't working as intended, flag it. If a step fails but the app is actually working as intended, flag that too. Report anything suspicious, and let a human decide."
"That one reframe changed the agent from a script-runner into something that felt more like a junior tester. You can't trust an agent you can't audit. Trust comes from the harness around it, not the model inside it."
This is the central insight running through the entire conversation: the agent isn't the product. The harness is the product. The system of context, boundaries, and human oversight that wraps the agent determines whether it's useful or dangerous.
The fear that agentic AI replaces testers is persistent. Huy's answer is specific about what changes and what becomes more valuable.
"Two things get more important: judgment and critical thinking."
"Judgment is knowing when the agent's output is trustworthy and when it isn't. The QE stops being the person who executes steps and becomes the person who reviews reasoning. You're not grading a test run. You're evaluating whether the agent understood the problem it was trying to solve."
"Critical thinking is the stress-testing instinct. Peter Pang said this well in his essay on AI-first engineering: 'the ability to criticise AI will be more valuable than the ability to produce code.' Same is true in QA. The next wave of great QEs are the people who can argue with the agent, find the gap in its reasoning, and spot the failure mode it missed. They're harness architects, not button-clickers."
He's also direct about an anti-pattern he sees spreading.
"QEs asking the agent to generate automation code for them. That's not agentic QA. That's just faster script-writing on the same broken treadmill. If the deliverable at the end of your week is still a pile of Selenium, the tool changed but the model didn't. You're still chasing coverage. You're still maintaining brittle artifacts. The shift isn't 'write the same thing faster.' The shift is 'stop writing that thing at all, and spend your time on what the agent can't do.'"
For QA leaders thinking about how their teams need to evolve, the message is clear: the role levels up, but only if you let go of the old deliverables.
The Test Runner failure story illustrates a principle Huy keeps returning to. He describes the difference between directing every action and defining the boundaries as the single most important design decision in agentic QA.
"The first version of that agent was directed. We told it exactly what steps to take, and it executed them. The result was an agent that optimised for completion, not for truth."
"What we changed wasn't the instructions. It was the boundary. We told the agent: you are skeptical by default. Your job is to understand the intent of the test, not to blindly follow the steps. If a step passes but the app isn't working as intended, something is sus. Flag it. If a step fails but the app is actually working as intended, that's also sus. Flag it. Report everything suspicious. Humans verify and decide."
"Those are boundaries. They tell the agent what 'done' and 'wrong' look like. They don't tell it how to get there."
The principle he leaves QA leaders with is simple: "If you tell the agent every step, you've written a script with extra steps. If you tell it what 'done' and 'wrong' look like, you've built an agent. That's the whole shift. You stop trying to encode the path. You start encoding the destination and the rules of the road."
The conversation around agentic QA doesn't exist in a vacuum. It's being forced by a velocity gap that's already opened up on the development side.
"Gartner says 25% of Y Combinator companies have 95% or more AI-generated code," Huy notes. "Google's CEO said over a quarter of new code at Google is AI-generated. By 2028, Gartner expects 40% of new enterprise production software to be vibe-coded. The dev side of the equation has already changed."
What happens to teams that don't close the gap? Two things, and both are bad.
"One: testing becomes the new bottleneck, ten feet downstream from the old one. Dev ships a feature in two hours. QA takes three days. Engineering is idle waiting for sign-off on work that's already done. You haven't solved the pain; you've just moved it, and now the people who used to enable velocity are the people blocking it."
"Two: quality debt compounds silently. Nothing visibly breaks for six months. The product feels fast, shippy, healthy. Then the regressions cascade, and you spend a quarter firefighting instead of shipping. The scariest version is the one where you don't even notice, because your existing test suite still passes. It's just that the test suite was written for a product that no longer exists."
As Huy puts it, borrowing from Pang: "Fast AI without fast validation is fast-moving technical debt." Validation has to move at the speed of implementation, or the system tears itself apart.
The rise of "vibe coding," where developers collaborate with AI through natural language and intuition rather than writing every line by hand, creates a specific problem for quality assurance. And Huy argues it comes down to one word: context.
"When code is written by a human, the human's intent is embedded in the code itself. In the structure, the comments, the naming, the commit history. When code is written from a natural-language prompt, the intent lives in the prompt, which is often disposable, or in the head of the person who wrote it. The spec evaporates. The code exists, but the why doesn't."
The defence against this is wrapping the agent in as much context as possible. Code, design, requirements, user stories, bug history, release notes, support tickets. The more context the agent has, the better it can define what the intent was supposed to be, and the better it can verify whether reality matches.
"Agentic QA lives or dies on context. Starve the agent of design, requirements, and history, and you get a guesser. Feed it everything, and you get a tester. That's the one-line version."
"And that's the harness. The harness isn't the agent. The harness is the system of context and boundaries around the agent that lets it reason. Build that, and vibe-coded code becomes testable. Skip it, and every prompt is a roll of the dice."
Huy led the development of Scout, an AI quality companion that launched on AWS's agentic AI technologies. Rather than a feature walkthrough, we asked him about the problem that started it all.
"The itch was a question I kept coming back to: how do you educate non-testers about quality? Most of the people building software today don't have a QA background. They ship, they ship fast, and they don't always know what they don't know. Scout started as my attempt to put a quality advocate next to every builder, even the ones who'd never written a test case in their life."
What surprised him was the reaction. "Most of the users we onboarded were technically advanced people. Developers, PMs, founders. Not junior. And they were genuinely surprised at the issues Scout surfaced in their own products. That was the delight."
He's also honest about the limits. "Agents are powerful, but they are not one-size-fits-all. If your app is very complex, or very old, or has a lot of undocumented state, the agent has a much harder time. And you have to be aware of the generic limitations of today's models: context windows, token limits, hallucinations. These are real. They don't disappear because the demo looked good. Part of our job as builders is to be honest about where the agent struggles, so the QA leaders adopting this don't walk in expecting magic and walk out feeling burned."
We asked Huy to look ahead. What does the QA function look like in organisations that embrace agentic testing versus those that don't?
The teams that embraced it: "Small, senior teams with enormous leverage. The QE-to-dev ratio drops, but the role level goes up. You're looking at harness architects, not button-clickers. Quality becomes continuous, not a gate. And interestingly, QA stops being a function and starts looking more like a discipline, the same way SRE happened to ops. A central group owns the harness, the evals, the boundaries. Every product team has agents doing the work inside that harness. The org chart reshapes."
The teams that didn't: "They look like the bottleneck everyone routes around. Engineering ships to staging, the AI-native teams ship to prod, and the old QA org is left gatekeeping a process no one waits for anymore. Underneath that, they're drowning in maintenance debt. The scripts compound, the coverage never quite catches up, and the team spends more of its time keeping the automation green and less finding bugs. Then even more. Then all of it."
"I want to be honest about the tone, though," he adds. "The gap is widening, but it's not a death sentence. There is still a path forward for teams that start moving now. Eighteen months is enough time to build a harness, move your team up the value stack, and be in the first group. It is not enough time if you wait twelve months and then start. That's the real message."
We ended where most readers will want to begin: what do you actually do on Monday morning?
"You don't need a transformation programme. You need one experiment, and a paragraph."
"Pick one repetitive, low-risk flow. A regression suite everyone ignores, a pre-release smoke test, a boring checklist. Put an agent on it for a week. Don't rip out your framework. Don't retrain the team. Watch what the agent catches, what it misses, and what your team does with the output. You'll learn more in five days than in five months of planning."
"But before you point the agent at anything, write down what 'quality' means for that flow in plain English. Not test cases. Not acceptance criteria. A paragraph that says, 'this is what good looks like, and this is what broken looks like.' If you can't write that paragraph, no agent will pass the test, because you haven't told it what the test is."
"Ninety percent of the difficulty of agentic QA isn't the agent. It's realising that nobody ever wrote down what 'quality' actually meant, and the agent exposes that gap immediately."
One flow. One paragraph. One week.
And the closer Huy leaves us with: "The agent isn't the product. The harness around it is. Build the harness, and start this week."
Huy Tieu is a Senior Product Manager at Katalon, where he leads Scout QA. He started his career as a Test Engineer and has held product roles at KMS Technology and Dwarves Foundation. Huy also published his own condensed take on this conversation on his personal blog, including diagrams on the trust loop, coverage ceiling, and velocity gap. You can find him on LinkedIn.
To learn more about how agentic QA is reshaping quality engineering, explore Katalon's Agentic QA blog series or see how Katalon True Platform brings agentic testing to the full quality lifecycle.
For the other side of this conversation, a QA Manager's view on making agentic QA work for real client teams, read Quality People: Gastón Marichal on Why Agentic QA's Hardest Problem Isn't the Technology.
And for a look at that same harness built as working code, see how Huy built a Playwright AI agent that hands Claude Code an entire QA job.