The denominator point is the one I would carry furthest, because there is a version of it that shows up before any tool is involved.
At the small end of ecommerce the platform would be measuring a site that is not eligible in the first place. Of 197 online stores I reviewed this summer, 52 had indexation as the single biggest problem on the site: filter, facet and pagination URLs picked up instead of the product pages. A dashboard pointed at one of those is not reporting low visibility. It is reporting an absent catalogue, and it will report it as a score that moves.
That is also why I read your 57 percent split as one objection rather than two at that size. "I do not believe the number" and "I cannot connect this to money" both resolve into the same question from an owner: would this number change what I do on Monday. When the answer is a template fix that has nothing to do with the measurement, the tool loses the argument before trust ever comes up.
On the response rate, your closing finding is the one I would want tested. One in 4,400 is close to what I get when I ask store owners anything about AI search. They are not resisting it. It has not reached them yet.
The denominator point is the one I would carry furthest, because there is a version of it that shows up before any tool is involved.
At the small end of ecommerce the platform would be measuring a site that is not eligible in the first place. Of 197 online stores I reviewed this summer, 52 had indexation as the single biggest problem on the site: filter, facet and pagination URLs picked up instead of the product pages. A dashboard pointed at one of those is not reporting low visibility. It is reporting an absent catalogue, and it will report it as a score that moves.
That is also why I read your 57 percent split as one objection rather than two at that size. "I do not believe the number" and "I cannot connect this to money" both resolve into the same question from an owner: would this number change what I do on Monday. When the answer is a template fix that has nothing to do with the measurement, the tool loses the argument before trust ever comes up.
On the response rate, your closing finding is the one I would want tested. One in 4,400 is close to what I get when I ask store owners anything about AI search. They are not resisting it. It has not reached them yet.
Very cool! Kudos!
Not only for this article but for future ones, I think there is value in you sharing ways people are wrong about many AI metrics assumptions.
As I am refining my Help Architecture tutorial, knowing how question and answer variation occurs in natural language is extremely useful.
I use AI as one tool. I don’t use it to generate thousands of question-answer pairs. My process has always been human centered and human curated.