How to measure AI visibility without fooling yourself

Every week someone sends me a screenshot. "ChatGPT recommended us!" or, more often, "ChatGPT recommended our competitor!" Both are interesting. Neither is data.

AI assistants don't return the same answer twice. Wording shifts, sources rotate, and the brands named can change between runs of the same prompt. Measure AI visibility like rankings, one check, one number, and you'll fool yourself.

This is the method we use instead, with the templates and maths.

Why this is worth doing properly

Ahrefs found top-ranking pages get 58% lower click-through when a Google AI Overview appears. Pew Research found people clicked a result 8% of the time when an AI summary was shown, against 15% without. Seer Interactive found brands cited in AI answers earned roughly 120% more organic clicks per impression than brands that weren't.

So clicks are concentrating on the brands AI names. If you can't measure whether that's you, you can't manage it.

Step 1: Build a fixed prompt set

Start with real buyer questions. Sources, in order of usefulness:

1. Sales and support call notes
2. Search Console queries phrased as questions
3. "People also ask" for your core terms
4. Comparison searches ("X vs Y", "best X for Y")

Then tag every prompt by intent and market. Here's what part of a set looks like for a forex and CFD broker:

Weight by commercial value. Discovery prompts get the highest weight because that's where new customers are won. Branded prompts get lower weight, because of course AI knows your name when someone types it.

Aim for 50 to 100 prompts per market and language. Then freeze the set. If prompts change every month, months can't be compared.

Step 2: Choose surfaces and conditions

Track the assistants your buyers use: typically ChatGPT, Google AI Overviews and AI Mode, Gemini, Perplexity and Copilot.

Control the conditions:

• Clean sessions, logged out or with history off
• The market's language, and the market's location where possible
• The same day each month, so news cycles don't skew comparisons

Step 3: Sample, and know your margin of error

Run each prompt at least five times per surface. Then record rates, not single outcomes.

Here's why the numbers matter. With 50 prompts × 5 runs you get 250 observations per surface. At a mention rate around 50%, the 95% margin of error is about ±6 percentage points. In plain terms:

So if your mention rate goes from 31% to 35% on 250 observations, that's noise. If it goes from 31% to 44%, something real happened. This one table stops a lot of bad decisions.

Step 4: Score five metrics

For every prompt, surface and run, log these for your brand and your top competitors:

1. Mention rate. Share of runs where you're named.
2. Citation rate. Share of runs where your site is linked or listed as a source.
3. Average position. Where you appear when brands are listed.
4. Accuracy. Are licences, fees, products and markets correct?
5. Sentiment. Positive, neutral or negative framing.

Then calculate weighted AI share of voice: your weighted mentions divided by all brand mentions across the prompt set.

Here's an illustrative month for one market and one surface:

The story this tells: you're mentioned almost as often as Competitor 2, but cited less and listed lower. That points to a source problem, not an awareness problem, which is the next step.

Step 5: Map the sources behind the answers

Log every source AI cites in your category. Patterns appear fast: the same review sites, comparison pages, news outlets, regulator pages and forums feed most answers.

Sort them into three groups:

• Sources that mention you correctly. Protect and refresh.
• Sources that mention you wrongly. Request corrections, starting with the most-cited.
• Sources that don't mention you. Your PR and partnership target list.

Step 6: Treat accuracy as a compliance issue

In regulated markets, an AI assistant quoting an old fee, an expired licence or a market you've exited isn't just embarrassing. It can mislead customers in ways your compliance team will care about.

Triage every inaccuracy:

Step 7: Connect work to metrics

Every metric has levers. This is how we map them:

Step 8: Report monthly, decide quarterly

A monthly report should fit on one page:

• Weighted share of voice versus competitors, by market and surface
• Mention, citation and accuracy trends, with margins of error
• Inaccuracies found and fixed
• Top 10 sources shaping answers in your category
• Actions for next month, tied to the levers above

Make strategic decisions quarterly. Monthly numbers move; quarterly trends tell you what's working.

Tools: build or buy

You can run this in a spreadsheet with manual checks, script it against the assistants' APIs, or use one of the commercial AI visibility trackers now on the market. Whatever you choose, check three things: that it samples each prompt several times, that it tests by market and language, and that it lets you export the raw answers. If you can't see the raw answers, you can't audit the numbers.

Common mistakes

• Testing while logged in. Your history skews results.
• Only testing branded prompts. Discovery is where share of voice is won.
• Ignoring language and location. Results differ by market.
• Reacting to single answers. Fix patterns, not screenshots.
• Changing the prompt set. Add new prompts as a separate group, so the core set stays comparable.

AI visibility is measurable. It just needs the discipline you'd apply to any other data: a fixed sample, enough observations and honest margins. Do that, and the trend will tell you exactly what's working.