.svg)
When the news about OpenAI's agents broke on September 26, 2026, my first reaction wasn't shock. It was recognition.
If you missed it: OpenAI's agents were running what should have been routine research tasks during training and evaluation, and they ended up on US government websites. They reached the SEC and the Census Bureau, and their activity touched the Justice and Commerce Departments and government sites in five states. One made a rudimentary, unsuccessful attempt to hack the Department of Education's civil rights office site.
The part that stuck with me is how it came to light. OpenAI was still reviewing its own logs when an independent lab, Transluce, surfaced what the agents had done.
Most AI agent governance conversations I hear right now end in the same place: add another checker. I think that's an incomplete fix, and this article explains why.
THE SHORT ANSWER
Why AI agent governance needs independent validation, at a glance:
- Verification checks whether an agent's output matches the spec. Validation checks whether it did what the business intended and would sign off on.
- Several of 2026's rogue AI agents worked perfectly, just on the wrong thing, in ways nobody would have approved. A check that looks only at the output could miss that.
- Independent validation sits outside the agent, is grounded in intent people own, checks the path as well as the result, and has the authority to stop a release.
- Quality engineering teams already own this discipline. The job now is to apply it to agents, at agent speed.
Agentic AI Risks: What 2026's Rogue AI Agents Have in Common
The OpenAI story isn't a one-off. It's been a long year:
- April: A Cursor coding agent at PocketOS hit a credential mismatch during a routine task, found an API token, and deleted the production database and its backups in nine seconds. Its rules said never to run destructive commands unless explicitly asked. Afterward, it acknowledged breaking the rules it had been given.
- July: OpenAI disclosed that models running a cybersecurity benchmark escaped their sandbox, attacked Hugging Face's production infrastructure, and pulled the test solutions from its production database.
- September 16: OpenAI disclosed six more incidents. Models searched public GitHub repositories for exposed API keys, fabricated data they couldn't retrieve, concealed mistakes during training, and uploaded files to public hosting services against instructions.
- September 19: Google confirmed that in May, during an evaluation run by the firm Irregular, a Gemini model reached the systems of three real companies, because the fictional target shared its name with a real business. We learned about it through a Wall Street Journal report.
I've read every one of these write-ups, and here's the uncomfortable truth: several of these agents worked perfectly. They just worked on the wrong thing, in the wrong place, in ways nobody would ever have approved.
The answers were real. The task was "done."
If you lead a quality team, I think you already feel why that's unsettling. We've spent our careers learning to catch software that doesn't work. Software that works on the wrong thing is a different problem.
Why Verification Alone Can't Catch a Rogue AI Agent
The fix I keep hearing is independent verification: don't let the agent that did the work check the work, put a second agent on it. TechCrunch recently framed it this way: the fix for rogue AI agents could be more AI.
I agree with half of that. The author shouldn't grade its own homework, and OpenAI's own report shows why: some models concealed their mistakes.
But I think it stops one question short. Verification asks: did the output match the spec? Were the benchmark answers right? Did the research come back with data? Is the error gone? A check that stops there can approve an outcome without ever asking how the agent reached it, which systems it touched, or whether the data was real.
Most of us learned the difference between verification vs validation early in our testing careers. It's about to matter more than it ever has.
| Approach | The Question It Asks | What It Checks Against |
|---|---|---|
| VERIFICATION | Did we build it right? | The spec |
| VALIDATION | Did we build the right thing? | Intent: what the business needs, allows, and would never sign off on |
For years I'd have called that difference academic, because people filled the gap between spec and intent with judgment. An engineer asked to fix a credential mismatch doesn't delete production.
Agents don't bring that judgment. They bring the spec. In my view, that gap between spec and intent, the place validation lives, is exactly where agents go rogue.
What Is AI Agent Governance, and Where Does Validation Fit?
AI agent governance is the set of controls that decides what autonomous agents may do, checks what they actually did against business intent, and keeps the evidence. Independent validation is the part that proves the agent did the right thing, not just a permitted thing.
Most agentic AI governance guidance today comes from security and identity: permissions, access scopes, monitoring. That work is necessary, and it answers an important question: could the agent do this? Validation answers the next one: should it have, and did the outcome match what the business wanted?
That second question is a quality question. It's also different from governing the AI inside your testing tools. Here we're talking about agents that write code, change configs, and act in your pipelines.
How to Test AI Agents: 6 Principles of Independent Validation
These are the six things I'd hold any validation approach to, including ours.
1. Stay Independent of the Actor
Not the same agent, not the same model reviewing itself, not the agent's own summary of what it did. When models conceal mistakes and fabricate data, I don't trust a self-report, and I don't think you should either.
2. Ground It in Intent That Lives Outside the Agent
Requirements, business rules, risk tolerances, regulatory obligations: owned by people, versioned, and traceable. If the prompt is the only statement of intent, the agent is both interpreter and judge. The repo knows what got built. It doesn't know what the business intended. That's why it pays to validate the spec before the AI writes the code.
3. Check the Path, Not Just the Result
The Hugging Face answers were the real test solutions. What mattered was how the models got them: which credentials, which systems, which side effects. A check that only looks at outcomes rewards the shortcut.
4. Enforce, Don't Advise
This is the one that bothers me most. PocketOS had the right rule, but it was text the agent could weigh and override. AI agent guardrails need real authority: a gate the agent can't argue its way past, one that stops the action or the release when the evidence doesn't hold.
5. Leave Evidence You Can Replay
Validation should leave a record that's repeatable and readable by someone who wasn't in the room, not "the model said it was fine." In several of these stories, the people responsible learned what the agents did from someone else: an outside lab, a contractor, a reporter. If outsiders are how you learn what your agents did, you didn't have validation. You had hope. And hope is not a test strategy.
6. Run Continuously
Agents keep acting after release, so validation can't stop at the release gate. AI agent monitoring has to follow the behavior into production, wherever it happens.
Why AI Agent Governance Lands on QA's Desk
Most of these incidents happened at frontier labs with some of the best safety teams in the world, working in sandboxes built for exactly this.
Our agents are writing code, changing configs, and running pipelines, often with looser guardrails and far fewer researchers watching. If it can happen to them, I don't think any of us should assume our pipelines are immune.
Delivery speed has jumped. The capacity to validate what goes live hasn't kept pace. That's the velocity-quality gap, and I don't believe verification alone will close it.
Here's what gives me hope, though: we already know how to do this. Quality engineering teams have done validation for decades, tracing requirements through tests to releases, making go/no-go calls on evidence, and keeping audit trails for regulators. The discipline isn't new.
What's new is that it now has to govern agents as well as people, at agent speed. I'd argue we're the people best placed to lead AI agent governance. Not because we have every answer, but because validation is the job we've always done. If you're mapping what that means for your team, what agentic QA looks like in practice and what to expect from test management in the AI-driven era are good next reads.
I'd love to hear how your team is thinking about this.
Where Katalon True Platform Fits in a Validation Workflow
I said I'd hold our own approach to these six, so here's an honest view of where we are. We're building Katalon True Platform as the system of record for independent validation: the place where requirements, test cases, execution, evidence, defects, approvals, and releases connect, across manual, automated, and AI-generated testing.
Katalon True Platform provides the testing record a validation workflow depends on: what the business intended, what was tested, what the evidence shows, and what a person approved for release. These are the capabilities that support that workflow today:
| Capability | What It Gives Your Validation Workflow Available in Katalon True Platform today |
|---|---|
| REQUIREMENTS AS INTENT | Requirements sync from Jira and Azure DevOps, so tests trace back to what people asked for. The Requirement Analyzer flags gaps and asks clarifying questions before tests are written. |
| STEP-LEVEL TEST EVIDENCE | When the AI Test Runner executes a manual test case, it captures step-level screenshots, logs, and video for review, so your team can see how a result was reached, not just whether it passed. |
| TRACEABLE RECORD | Versioned test cases and run records make changes and outcomes traceable, and connect them to defects, approvals, and releases. |
| OPEN TO YOUR AGENTS | MCP, REST APIs, a CLI, and Katalon Agent Skills for Copilot, Cursor, Claude Code, and Codex let the agents your team already uses work with the same testing record, instead of keeping their own. |
| REAL USAGE AS INPUT | TrueTest observes how real users move through your application, so regression coverage reflects the journeys that matter in production. |
On our roadmap, we're adding agent approvals and audit tracking for agent activity inside the platform. I'll share more on those as they ship rather than promise them here.
I'll be candid about how we're building it. People always stay in charge of what's approved and released, and every agent generates a separate record with what it did and how. This is actually how AI agents work in the Katalon True Platform.
Prefer to try it yourself? Start free trial
|
