Why ChatGPT gives different answers to the same question

ChatGPT gives different answers to the same question because each answer is sampled, not looked up. Its searches and chosen sources can also change between runs, and memory can shape replies for each person. So one check tells you little. To learn whether ChatGPT names a brand, ask many times and report a share, with a range around it.

The best public evidence is a SparkToro and Gumshoe study from January 2026. Volunteers ran 12 prompts about 60 to 100 times each, for 2,961 responses. The chance of getting the same brand list twice in 100 runs was under 1 in 100. The study isn’t peer reviewed, it covers models from late 2025, and one co-author works at an AI tracking company.

What makes ChatGPT’s answers change between runs?

ChatGPT’s answers change because several steps can each go differently. No operator publishes how much each one contributes, so the list below shows what’s documented, not a ranked breakdown. Some of these apply to one person asking twice. Others explain why two people see different things, or why an answer changes over a month.

Sampling. OpenAI’s documentation describes its APIs as non-deterministic by default. Its seed setting makes a best effort at repeatable output, and OpenAI says determinism isn’t guaranteed. That page covers the API, and ChatGPT itself offers no such setting that we could find.

Search and source choice. Ahrefs, which sells AI visibility tracking, followed about 37,500 cited websites for 20 days in a study published in October 2026. When ChatGPT’s searches didn’t repeat, cited sites changed 91% of the time, compared with 81% when they did. Ahrefs reads this as much of the churn happening after search, when sources are chosen. It’s a correlation, and it comes from a seller of tracking.

Personal context. OpenAI’s help page says Memory can draw on past chats, saved memories, custom instructions, saved files and connected apps, depending on plan and region. Google has its own version. AI Mode can reference saved search history for adults with history and personalised recommendations turned on, and its memory feature is offered in the US, in English.

Wording. The SparkToro study asked 142 people to write their own prompt for one request. The average semantic similarity between prompts was 0.081, so real questions rarely match each other. A tracked question is one version of many.

Which product you use. Google says AI Mode and AI Overviews may use different models and techniques, so their responses and links vary. Models also change over time, which is why a result from last quarter may not hold today.

Is anything stable across ChatGPT’s answers?

Yes. Which brands appear tends to be steadier than their order or the pages cited. In the SparkToro study, City of Hope appeared in 69 of 71 ChatGPT responses about West Coast cancer care, yet ranked first in only 25. Its authors conclude that ranking position is too random to track, and that visibility across many prompts and runs seems a reasonable, imperfect measure.

Ahrefs saw a similar split. About 50% to 70% of cited websites appeared on only one day, compared with about one in five brand names. Stability also varies by category. In the SparkToro work, leading headphone brands appeared in 55% to 77% of 994 responses, while Gumshoe’s separate analysis put brand design agencies in the 30s and 40s. Crowded categories with no clear leaders may simply vary more.

How stable are the sources AI assistants cite?

The sources AI assistants cite are far less stable than most reports imply. In Ahrefs’ study, 59% of the websites ChatGPT cited on day one were gone by day two, and 4% lasted all 20 days. About 75% of the citations that dropped came back within the 20 days, so a lost citation is often temporary. Treat one day’s drop as noise until it lasts.

How many times should you ask ChatGPT the same question?

Ask as many times as your decision needs, because each extra run narrows the range around your result. No published study sets a correct number. The arithmetic does give a guide, though. If ChatGPT names you in 60% of answers, five runs can’t tell you much, since the true rate could plausibly lie anywhere from 23% to 88%.

This changes how you should read a report. Going from 40% to 60% on 20 runs of one question may be no change at all, because the ranges overlap heavily. A real shift should show across many questions and hold over several checks. It helps to track questions you didn’t work on as a control, so you can see whether everything moved together.

Why can’t you just check ChatGPT yourself?

You can check ChatGPT yourself, but one answer in your own account is a poor sample. It’s a single draw, shaped by your memory and past chats, using your own wording. Your reply may name your brand partly because you’ve discussed it there before. Use a temporary chat or a clean account where you can, and treat what comes back as an example, not a measurement.

A check by hand is still worth doing. Choose ten questions a buyer would really ask, run each several times, and write down who’s named and which pages are cited. Paste the answers into our free AI answer analyser to count them. The AI question generator can suggest more questions.

It gets hard when you need to repeat that across engines every month. Quotable does this by asking each question several times on each engine, and it shows ranges with its results. Ahrefs Brand Radar, Semrush’s AI Visibility Toolkit and Gumshoe also measure answers. Tools like ours ask through the assistants’ APIs, so we can’t see any one person’s memory, and SparkToro’s authors note it’s still an open question how closely API answers match what people see. Nobody can guarantee that an assistant will name you, and repeat runs only make the measurement fairer.

How to explain this to your team

A note you can adapt: “AI answers vary from run to run, so a single screenshot isn’t evidence. We report how often we’re named across repeated runs, with a range. We don’t report rank positions, because order is unstable. We treat a change as real only when it shows across many questions and holds over more than one check.”