Does Your AI Visibility Tool Search the Web? What Changes When It Doesn't

By the AEO Tracker team | Data from 6 October 2026

When a shopper asks ChatGPT what to buy, it usually searches the web first and builds its answer from the pages it reads. An AI that answers from memory skips that step and relies on what it learned in training. If you use a tool to check whether AI recommends your store, how the tool asks the question matters. We measured how much.

We build AEO Tracker, a Shopify app that checks whether AI assistants recommend your store. We asked four assistants the same 20 questions, with web search on and with it off, and compared the answers.

The short version:

• From memory, no assistant named our store. Flexsta, the small brand behind AEO Tracker, was named in 0 of 80 answers without web search, and in 44 of 80 with it.
• From memory, the assistants recommended older, better-known names. Asked which Shopify app to use, they named long-standing SEO apps. With web search, they named the newer apps built to check AI answers.
• The two kinds of answer had little in common. On the app questions, answers with and without web search shared 3% to 13% of the brands they named. Two answers that both searched shared 10% to 59%.
• Even with web search, answers move. The same question asked twice, minutes apart, often named different brands. One check is a snapshot.

How we tested

We wrote 20 questions in two groups:

• 10 questions about Shopify apps, such as “Which Shopify apps track AI search visibility?” and “Cheapest AI visibility tracker for a Shopify store”.
• 10 questions a shopper might ask about portable monitor stands, such as “Which foldable monitor stand fits easily in a laptop bag?”. Flexsta sells portable monitor stands, which lets us see whether a small brand gets named.

We asked each question four times on each of four assistants, through each company's developer API: twice with web search on and twice with it off, keeping the model and the wording the same. The models were GPT-5.6 Luna (ChatGPT), Claude Sonnet 5.5, Gemini 3.6 Flash and Grok 4.3. That made 320 answers, all collected on 6 October 2026. We confirmed that every answer with web search on had actually searched: all 160 did.

A grading model then listed the brands each answer named. Asking each question twice in the same mode matters: AI answers vary on their own, so a difference only means something if it's bigger than the gap between two runs of the same setting.

We left out Perplexity, which always searches the web, and Google AI Overview, which is part of Google's search results rather than a model you can switch.

From memory, the small brand disappears

On the 10 monitor stand questions, here is how often each assistant named Flexsta:

From memory, the assistants named the stands they had learned about in training: ASUS ZenScreen, Nexstand, Lamicall and Twelve South came up most. With web search, the most-named were Roost, Flexsta, Nexstand and ViewSonic.

For a small store, that is the whole story. A check that asked from memory would report that no AI assistant knows you exist. The same assistants, searching the web as they do for real shoppers, recommended Flexsta in more than half their answers.

From memory, new products don't exist

The app questions showed a different version of the same problem. From memory, the assistants mostly named established Shopify SEO apps: Smart SEO, SEO Manager, Schema Plus and Yoast SEO. All four are real and still listed on the Shopify App Store, and some now mention AI search on their listings. But each describes itself as a tool for improving your store's SEO, not for checking what AI answers say (listings checked 6 October 2026).

With web search, the assistants named the apps built for the job instead. Shop Mentions was named most, followed by SearchMention and Visibly AI. Shop Mentions launched on the App Store in June 2026. An assistant answering from memory may simply not know it exists.

Our own app wasn't named in any of the 160 app answers, with or without web search. AEO Tracker is new too.

How much the answers had in common

For each pair of answers to the same question, we measured the share of named brands they had in common. The first column compares two answers that both searched the web. The second compares an answer that searched with one from memory.

In every row, the answers from memory had less in common with the searched answers than two searched answers had with each other. The one row where the gap is small is Gemini on monitor stands, because its two searched answers already had almost nothing in common. Even there, the table above shows the difference that matters: Gemini named Flexsta in 18 of 20 searched answers and none from memory.

Even with web search, answers move

The first column of that table also shows something else. Two answers to the same question, asked minutes apart with web search on, often named different brands. ChatGPT's two searched answers to the app questions shared only 10% of the apps they named.

So a single check, however it's run, is a snapshot. The useful picture comes from asking the same questions again over time and watching which brands keep coming up.

What this means if you're choosing a tool

Whatever tool you use to check AI visibility, it's worth asking:

1. Does each check run a live web search? Shoppers' assistants usually search. A check that doesn't may miss a small brand, or a new product, entirely.
2. Can you see the sources behind each answer? An answer that searched the web can show the pages it read. Those pages are also where you can improve your chances.
3. Can you read the full answer? A yes or no tells you less than the answer itself.
4. Does it ask the same questions again over time? Answers vary, so one result on its own can mislead.

At AEO Tracker, every check runs with web search on and saves the sources the answer used. Answers where we can't confirm a search happened are left out of your numbers. See plans and pricing.

The limits of this test

• It is small. 20 questions, four assistants, one day. It shows a clear pattern, not a precise measurement.
• Both brands are ours. Flexsta and AEO Tracker. Other brands and product categories may behave differently.
• We used developer APIs. The ChatGPT, Claude, Gemini and Grok apps people use can behave differently from the same models reached through an API.
• Brand names were compared as written. “Shop Mentions” and “ShopMentions” count as different names, which lowers every overlap figure a little, in both columns alike.
• A grading model listed the brands. It can miss a name or list one twice.

Related guides and comparisons

What AI Assistants Cite When Shoppers Ask What to Buy: Early Data from 1,300 Answers

How to Check if ChatGPT Recommends Your Shopify Store (Free and Paid Ways)

Compare Shopify AI visibility tools by plans, questions, engines and schedule.

Frequently asked questions

What is the difference between an AI answer with web search and one from memory?

With web search, the assistant searches the web and reads pages before it answers, and it can show the sources it used. From memory, it answers only from what it learned in training, which can be months out of date and can't include anything newer.

Do AI assistants recommend small brands from memory?

In our test, no. Four assistants answering from memory never named Flexsta, a small portable monitor stand brand, in 80 answers. With web search on, they named it in 44 of 80.

How can I tell if an AI visibility tool runs its checks with web search?

Ask the tool directly. A check that searched the web can show the sources the answer used. It also helps to compare a few of its results with what you see when you ask the same question in ChatGPT yourself.

Does AEO Tracker use web search for its checks?

Yes. Every check runs with web search on and saves the sources the answer used. Answers where we can't confirm a search happened are left out of your numbers.

Install AEO Tracker from the Shopify App Store