This week we pointed twelve AI agents at our own company and told them to break things. Audit every property we run: the search rankings, the chatbots, the phone agents, the demos, the email automations, the deploys. Assume everything is broken until proven otherwise, and bring back evidence.
They came back with a long list of problems, including several marked critical. And here is the part of the story I actually want to tell you, because it is the part every business owner experimenting with AI needs to hear: about half of the headline findings were wrong.
What the audit got wrong
One agent reported that a search feature on one of our sites was broken on phones and tablets. I re-ran the exact scenario in a clean browser: it worked on every screen size. The agent's own test harness had been fighting itself, and it read its own timeout as a product bug.
Another reported a customer-facing button missing entirely. It was there, visible, working. Another flagged a domain as a total outage; the domain was a documented not-yet-launched item sitting exactly where it was supposed to sit. Another described a cookie banner "covering" prices that, in the screenshot the agent itself had taken, were plainly visible above the banner.
None of these were lies. They were confident, plausible, well-written claims from a system doing its honest best, and every one of them collapsed the moment a second set of eyes reproduced the scenario. If we had acted on the report as delivered, we would have spent the day "fixing" working software, and I promise you at least one of those fixes would have broken something real.
What the audit got right
Now the other half. The same audit found that one of our own showcase sites had a completely dead chat assistant, silent to every visitor while every monitoring light glowed green. We wrote that one up in detail. It found a cleanup job that had been quietly doing nothing for two months while reporting success. It found a public demo making a promise to visitors that the system was not keeping, and a stale price on one of our own pricing pages.
Every one of those was real, reproduced, fixed the same day, and re-verified on the live site afterwards. Several had been invisible for weeks precisely because they failed silently. No human review had caught them, and no human review was ever going to, because no human has the patience to click every button on eleven properties while distrusting every green light.
So the audit was simultaneously half wrong and the most valuable thing we did all week. Both facts matter.
The rule that makes AI audits work
The value of an AI audit is not the findings. It is the discipline that kills the false ones.
An AI finding is a claim, not a fact. Treated as a claim, it is cheap, tireless coverage no human team can match: the agents read everything, clicked everything, and surfaced real rot that had survived months of humans looking politely past it. Treated as a fact, the same report is dangerous, because AI systems deliver their wrong conclusions in exactly the same confident prose as their right ones, and a busy owner cannot tell the difference by tone.
The fix is not smarter AI. It is a standing rule between you and the machine:
- Nothing gets acted on until a human, or at minimum an independent second check, reproduces it on the live system.
- Every claim ships with its evidence: the screenshot, the failing request, the record that should exist and does not. No evidence, no ticket.
- The false positives get recorded, not just discarded. The patterns in what your AI gets wrong teach you where to distrust it next time, and that knowledge compounds.
That third one is underrated. After this audit we know our agents over-report failures on slow-loading pages and under-report failures that return a polite success code. That is now a permanent note in how we run the next one.
What this means if you run a small business
You do not need twelve agents. But if you are using AI to check anything that matters, your website, your books, your inbox, your reviews, the same two sentences apply at every scale: let it look at everything, and let it decide nothing.
The businesses that will get burned by AI in the next few years are not the ones that use it. They are the ones that take its word for things. And the businesses that will quietly pull ahead are the ones that pair the machine's inhuman patience with a human's stubborn insistence on seeing the evidence, which is, not coincidentally, how we build the AI that answers your phone: it captures everything and hands the judgment calls to you.
Distrust is not the opposite of using AI well. It is the method.
Frequently asked questions
Can I trust an AI audit of my website or business?
Trust it to look everywhere; do not trust it to be right. In our own twelve-agent audit, roughly half the headline findings were false alarms that collapsed under reproduction, while the other half were real, silent failures no human had caught. The audit is valuable exactly when every finding is treated as a claim requiring evidence, not a verdict.
Why do AI tools report problems that do not exist?
Because the AI's own testing environment fails in ways it misreads as product failures: timeouts, race conditions and stale assumptions all look like bugs from the inside. The AI then describes its mistaken conclusion in the same confident tone as a genuine finding. Reproducing the claim in a clean, real-world session is the reliable filter.
What is the right way to use AI for business operations?
Give AI the jobs that reward tirelessness, checking, reading, monitoring, capturing, and reserve judgment for humans. Require evidence with every claim, act only on what has been reproduced, and keep a record of what the AI got wrong so you learn where its blind spots are. Coverage from the machine, verdicts from you.