You left out the biggest question of all. Do you think that currently, you can accurately and reliably check brand mentions and track them over time? My answer is no.
These three biggest problems.
The “Segment of One”: Modern AI uses “dynamic profiles” that update in real-time based on the user’s browsing behavior, scroll depth, and even device type. This makes a single “rank” or “mention score” for a brand technically impossible to standardize. This is not like SEO or ranking in Google where there is actually a database to draw from. The difference between measuring AI visibility and Google rankings could not be further apart.
Synthetic Data Problem: Tools like Semrush and Nightwatch mostly rely on synthetic queries.These are automated bots asking a fixed set of questions via API. These bots do not have the user’s context, history, hobbies, or professional, which is why their “visibility scores” mostly differ from what real users see.
The “Transparency” Gap: OpenAI, Google, Anthropic & Perplexity AI do not provide analytics APIs for brand mentions. Until they do, any tool claiming to “track” mentions is effectively using “directional surveillance” or simulations rather than hard data.
So a lot of my questions on that survey are applying to where we are right now. No I'm not going to pay a lot for a tool that can't reliably track brand mentions. We don't have that data yet.
My problem isn't that people are trying to build it and who are informing people the limits of that data they are providing. But those that are selling it as an absolute that you can track brand mentions, I believe are either being misleading or don't actually understand the tool they created.
Chris, thanks for engaging with this seriously, and you're right on the technical realities. Non-deterministic outputs, personalized answers, synthetic-query methodology, and the absence of proper analytics APIs from the model providers are all real limitations, and anyone claiming otherwise is either overselling or hasn't looked closely at what their own tooling does under the hood.
Where I'd push back gently is on the step from "the data is imperfect" to "the category isn't worth paying for right now." That's a defensible position, and it's one I want the survey to be able to measure honestly, but it's also a values call on top of the technical facts, not a technical conclusion by itself, I think. Directional data with acknowledged limits is either useful or it isn't, depending on what you're doing with it, and the people doing this work every day will have different reads on that. That split is exactly what this survey is trying to surface.
You're also correct that the questions frame the current state of the industry rather than what it might look like when proper APIs exist. That's deliberate. I'm asking what practitioners think about the platforms available today, at today's price points, doing today's imperfect job. If the answer comes back "not worth it yet," that's a legitimate and important finding and I'll publish it that way.
Which is a long way of saying: your response is exactly the kind of technically grounded read the survey most needs. I appreciate you contributing! Thank you.
And just so you know I support people building these tools even if it didn't sound like it. I think that we are going to get access to more data and those who are building the tools now are going to be ahead of the curve. And the part about whether I find it useful right now is strictly from my personal POV not a recommendation to others.
My reasoning is different because I've built a tool myself and I've done a little research and I feel like I can get the directional signals myself.
My tool is for local SEO only. And I put more limitations on it on purpose. You type in a service area and a search term. It returns 10 brands that it recognizes for that.
No "can I help you more" no descriptions, no links. Nothing. Just see if your brand pops up. And even that is just a snapshot of that moment in time because the list can change in subsequent searches.
Man, I 100% agree with you on the building (I did, too) and those who do will be further ahead in 12 - 18 months (maybe that's self-serving?). there is just SO MUCH to understand with all of this new data. Sadly, when I see large brand sin the space telling us it's all the same work as traditional SEO and just doing that helps you win...that's disappointing. I've written about that a bunch, but I also recall similar statements and ideas floated during the earliest days of SEO and when the then-new tools started popping up. ;)
You left out the biggest question of all. Do you think that currently, you can accurately and reliably check brand mentions and track them over time? My answer is no.
These three biggest problems.
The “Segment of One”: Modern AI uses “dynamic profiles” that update in real-time based on the user’s browsing behavior, scroll depth, and even device type. This makes a single “rank” or “mention score” for a brand technically impossible to standardize. This is not like SEO or ranking in Google where there is actually a database to draw from. The difference between measuring AI visibility and Google rankings could not be further apart.
Synthetic Data Problem: Tools like Semrush and Nightwatch mostly rely on synthetic queries.These are automated bots asking a fixed set of questions via API. These bots do not have the user’s context, history, hobbies, or professional, which is why their “visibility scores” mostly differ from what real users see.
The “Transparency” Gap: OpenAI, Google, Anthropic & Perplexity AI do not provide analytics APIs for brand mentions. Until they do, any tool claiming to “track” mentions is effectively using “directional surveillance” or simulations rather than hard data.
So a lot of my questions on that survey are applying to where we are right now. No I'm not going to pay a lot for a tool that can't reliably track brand mentions. We don't have that data yet.
My problem isn't that people are trying to build it and who are informing people the limits of that data they are providing. But those that are selling it as an absolute that you can track brand mentions, I believe are either being misleading or don't actually understand the tool they created.
Chris, thanks for engaging with this seriously, and you're right on the technical realities. Non-deterministic outputs, personalized answers, synthetic-query methodology, and the absence of proper analytics APIs from the model providers are all real limitations, and anyone claiming otherwise is either overselling or hasn't looked closely at what their own tooling does under the hood.
Where I'd push back gently is on the step from "the data is imperfect" to "the category isn't worth paying for right now." That's a defensible position, and it's one I want the survey to be able to measure honestly, but it's also a values call on top of the technical facts, not a technical conclusion by itself, I think. Directional data with acknowledged limits is either useful or it isn't, depending on what you're doing with it, and the people doing this work every day will have different reads on that. That split is exactly what this survey is trying to surface.
You're also correct that the questions frame the current state of the industry rather than what it might look like when proper APIs exist. That's deliberate. I'm asking what practitioners think about the platforms available today, at today's price points, doing today's imperfect job. If the answer comes back "not worth it yet," that's a legitimate and important finding and I'll publish it that way.
Which is a long way of saying: your response is exactly the kind of technically grounded read the survey most needs. I appreciate you contributing! Thank you.
Duane
And just so you know I support people building these tools even if it didn't sound like it. I think that we are going to get access to more data and those who are building the tools now are going to be ahead of the curve. And the part about whether I find it useful right now is strictly from my personal POV not a recommendation to others.
My reasoning is different because I've built a tool myself and I've done a little research and I feel like I can get the directional signals myself.
My tool is for local SEO only. And I put more limitations on it on purpose. You type in a service area and a search term. It returns 10 brands that it recognizes for that.
No "can I help you more" no descriptions, no links. Nothing. Just see if your brand pops up. And even that is just a snapshot of that moment in time because the list can change in subsequent searches.
Man, I 100% agree with you on the building (I did, too) and those who do will be further ahead in 12 - 18 months (maybe that's self-serving?). there is just SO MUCH to understand with all of this new data. Sadly, when I see large brand sin the space telling us it's all the same work as traditional SEO and just doing that helps you win...that's disappointing. I've written about that a bunch, but I also recall similar statements and ideas floated during the earliest days of SEO and when the then-new tools started popping up. ;)