How to measure AI visibility honestly: sample size and margin of error

A single AI answer is one draw from a noisy process. Treat it like a poll, not a scoreboard — or you will chase changes that are just noise.

One answer is one sample

Ask the same question twice and the assistant may name different brands. That variance is normal — the answer is generated, not looked up. So any one check is a sample of size one, and a brand appearing (or not) in it proves very little on its own.

The honest unit is a rate over many prompts and repeats: “mentioned in 15% of 120 checks”, with the 120 shown. Report the number without the sample size and you are inviting people to over-read a coin flip.

When a change is real

Two checks apart in time will differ. Before you call that a trend, ask whether the gap is bigger than the margin of error for your sample size. A move from 14% to 17% on a small sample is usually noise; label it as such rather than dressing it up as growth. Answerly marks within-error changes “≈ noise” on purpose.

Things not to fabricate

Do not invent competitors the model only hallucinated (the fake “<word> LLC” firms cited through a maps link), do not report a citation you cannot point to, and do not promise placement. Surface the uncertainty instead. Numbers you can defend to a client beat impressive numbers you cannot.