The One Metric to Ask About Before Buying Into "AI Marketing"
The monthly report lands in the inbox. It gets opened at the kitchen table at eleven at night, after the kids are down and the van is unloaded and the day is finally finished. Impressions, up. Engagement, up. There is a line on a chart heading in the right direction, and a friendly note from the vendor about momentum building. The owner reads it twice, because reading it twice feels like the responsible thing to do, and because something on the page refuses to settle. Then the thought arrives that spoils all of it. The phone rang about as often last month as it rang the month before. The one job that got booked came from a neighbour who already had the number.
Nothing in that report is false. That is the strange part, and it is why the phone call to the vendor the next morning goes nowhere.
There is one question that will tell you more about a marketing vendor than any deck, any case study, and any demo. How do you count a result, and does that number correspond to an actual customer or a booked job? Ask it early. Ask it before the contract, before the onboarding call, before anyone says the word "strategy." A vendor who can answer plainly, in one or two sentences, without reaching for a dashboard, is measuring your business. A vendor who cannot is measuring their own activity and hoping the two look similar enough on a monthly report.
A result metric is a number that corresponds to a specific human being who contacted your business and can be traced to something you could put on an invoice. That is the whole definition. Everything else is context. Context is useful, and I will defend it before this is over, but context does not pay a subcontractor or cover a lease payment.
A vendor who cannot tell you how they count a result is not measuring your outcome. They are measuring their own effort and inviting you to feel good about it.
Why the impressive numbers are usually the wrong ones
Stand beside each of those numbers at the moment it gets made and you will see why they are so easy to produce.
An impression is a server logging a request. That is the entire event. Your content was served somewhere, to someone, possibly for a fraction of a second, and the log line does not care whether a human ever glanced at it.
An open is stranger still. The email arrives, and before any person has looked at anything, a mail client or a privacy proxy reaches out on the recipient's behalf and loads the tracking pixel. Sometimes a security scanner does it instead. The counter ticks up. Nobody opened anything.
Engagement is a bucket, and you should picture what gets tossed into it. A scroll. A hover. A thumb that brushed the screen inside a coat pocket. A competitor checking on you. A bot. All of that lands in one number that then gets reported as interest in your business. Views are the same story with a video attached. Something started playing. It may have played on mute in a feed that was already scrolling past.
None of these numbers are lies. That is exactly what makes them slippery. They are real measurements of real events, and they genuinely do correlate with activity. The trouble is the leap from activity to money, which is precisely the leap the report quietly declines to make. It leaves that one for you, at the kitchen table, at eleven at night.
I spent fifteen years in software engineering before fifteen years pastoring in Reformed churches in Ontario, and I have come back to engineering to marry the two. Both careers taught me the same lesson from opposite directions. You get the behaviour you measure. Count sermons preached and you will preach a great many sermons. Count lines of code and the codebase will grow beautifully. Count impressions and you will end up with a vendor who has become genuinely, impressively excellent at generating impressions.
Vanity metrics versus metrics that mean something
Here is the comparison I would want a small-business owner to keep in their back pocket.
Vanity metrics a vendor might report:
- Impressions. Your content was served somewhere, to someone, possibly for a fraction of a second.
- Opens. An email was rendered, or a privacy proxy rendered it, or a security scanner did.
- Engagement. A composite of taps, scrolls, hovers, and clicks that can move for reasons that have nothing to do with buying intent.
- Views. A video started playing. It may have played on mute in a feed that was already scrolling past.
- Followers or list size. A number that grows whether or not anyone in it ever buys.
Metrics that actually mean something to a business owner:
- A real call from a real person who wanted something you sell.
- A booked appointment sitting on your calendar with a name attached.
- A quote request, a form submission, or a genuine reply from a human being, not an auto-responder.
- A closed job or a signed contract, with a date and a dollar figure you recognize.
- Repeat business and referrals from someone who first arrived through that channel.
Read the second list again and notice who is standing in it. There is a person in every line. That is not sentimentality, it is the test.
If you cannot point at a human, you cannot point at revenue.
What a straight answer sounds like
Put the question on the table and watch the vendor's face. A good one does not flinch, and the answer has the awkward parts left in. It sounds roughly like this:
We count a lead as a call that lasted longer than thirty seconds and came from a tracked number. We exclude calls from the same number within the same week. We cannot attribute the customer who saw your ad on Tuesday and walked into your shop on Saturday. Here is how we tell you that, instead of pretending otherwise.
That last sentence is the one to listen for.
Attribution is genuinely hard for small businesses. Phone calls, walk-ins, word of mouth, and the ad someone half-remembers are all tangled together in the same week, and they do not come apart cleanly. Any vendor who claims to untangle them perfectly is either running a model they have not explained to you or telling you what you want to hear. Honesty about the fuzzy edges is a stronger signal of competence than a clean number ever was.
The awkward part, and why I am writing this anyway
I run WiseAI Agency. We build AI front desks and websites for small businesses in Ontario, which makes me one of the vendors you might be evaluating. This is the standard I hold my own work to, and we are early, with a very small number of real paying customers. So I am not going to wave a portfolio-wide success rate at you, because I do not have one worth quoting, and I would rather tell you that than manufacture one.
What I can tell you is how we count. A result is a call that connected and was captured with the caller's name and reason. It is a message that a real person left. It is an appointment that landed on a calendar. It is a reply from a human. If a call goes to the AI and never reaches a person who could actually help, we do not get to count that as a win, because it was not one.
That same conviction is sitting underneath the product itself. The AI is a bridge to a person, never a substitute for one. It answers when you cannot. It captures what matters. It is honest about being AI. It hands the conversation to a human being as quickly as the situation warrants. A system built to replace the human would be easier to sell and worse for the customer on the other end of the line, and it would produce exactly the kind of metric that looks wonderful in a report and means nothing at all in a bank account.
Frequently asked questions
How do I know if my AI marketing is actually working?
Count the things with a person attached: calls, booked appointments, quote requests, replies from real humans, and closed jobs. Compare those counts to the same period before you started, and be honest with yourself about seasonality in your trade. If your vendor's report is full of impressions and engagement but your phone rings the same amount, the marketing is not working yet, whatever the dashboard says.
What metrics should a small business track?
Track the smallest number of things you will actually look at every month. For most local service businesses that is inbound calls, booked jobs, average job value, and where each new customer says they heard about you. Ask that last question at intake, out loud, every time. It is unglamorous, and it beats most analytics tools for accuracy.
Is it always wrong for a vendor to report impressions?
No. Reach and impressions are legitimate diagnostic numbers, and they help explain why results moved or did not. The problem starts when they are presented as the result rather than as context for it. A good report shows both and is clear about which is which.
What should I ask an AI vendor before signing anything?
Ask how they count a result, what they cannot attribute and why, who owns the data and the phone number if you leave, and what happens when the AI cannot help a caller. That last one matters more than people expect. A system with no clean handoff to a human will quietly lose you the calls that were worth the most.
Does honest measurement cost more?
Not inherently. Tracking a phone number, logging calls with names and reasons, and asking new customers how they found you are all inexpensive. Pricing varies by what you need built, and affordable monthly plans exist across the market. Ask any vendor for their pricing page rather than a number quoted in an article.
If you want a straight answer about your own numbers
If you are evaluating a vendor right now, take that one question into the meeting and see what happens to the room. You will learn more in thirty seconds than in an hour of slides.
And if you would like to talk it through with someone who will tell you plainly what he can and cannot measure for you, visit wiseaiagency.com and book a call. A real person will be on the other end. That is rather the point.
John Moelker is an engineer (15 years) and pastor (15 years), founder of ChurchWiseAI and WiseAI Agency.