ChatGPT API vs ChatGPT: Why AI Visibility Checks Don't Match What Users See

Here's a situation more marketing teams are running into. A report says ChatGPT recommends your brand for most of your category questions and cites your pricing page. Then a customer, or your CEO, opens chatgpt.com, asks one of those same questions, and gets a different list of brands and a different set of links.

Often nobody did anything wrong. The report may have been built from the OpenAI API rather than the ChatGPT app, and an October 2026 research preprint shows those are not the same thing, especially when it comes to which sources get cited.

This explainer covers what the paper found, why the API and the app give different answers, and how to read any AI visibility number with that in mind.

The short answer

The OpenAI API and the ChatGPT app can run related models, but they are different products. The app rewrites your question into its own searches, runs inside OpenAI's own interface and settings, and changes models on its own schedule. A developer calling the API chooses a model and settings. In this paper, the API calls had web search enabled and no system prompt; that does not make them a replay of the web app.

In the paper, Google results and the API together recovered roughly half of the cited sites, but only 26.2%–28.3% of the cited pages. That is better agreement at the site level, not proof that brand recommendations match. If your report tracks exact URLs from the API, treat it as a partial signal of the app's sources, not a copy of them.

What the research paper tested

In October, researchers at the National University of Singapore and Xi'an Jiaotong University posted a preprint, Understanding ChatGPT Citations, that looked at the ChatGPT web app from the inside. They recorded the data their own signed-in sessions received: the searches ChatGPT ran, the pages those searches returned, and the pages it picked as sources. Search Engine Journal's write-up covers the setup in detail.

They used 80 real questions from the public Search Arena dataset (40 in English, 40 in Chinese) and ran each one 20 times on GPT-5.3 Instant and GPT-5.5 Instant, mostly between March 29 and July 29, 2026. Then they checked whether two common stand-ins could recover what the app cited: Google's organic results (up to five pages per question) and OpenAI's Responses API with web search turned on, queried within three days of the app runs.

Even combined, Google and the API found only about a quarter of the pages ChatGPT cited: 28.3% for GPT-5.3 Instant and 26.2% for GPT-5.5 Instant. They did better at the site level, finding about half of the cited sites (50.3% and 48.7%). The authors sum it up in the abstract: both stand-ins "match ChatGPT Web better at the hostname and entity levels than at the exact-URL level."

Two more findings explain part of the gap:

• ChatGPT never searched the question as written. In GPT-5.3 Instant and GPT-5.5 Instant runs where the searches were visible, ChatGPT always rewrote the question into its own search terms. For Chinese questions, about two-thirds of those runs searched only in English.
• The app disagrees with itself. For GPT-5.5 Instant, 44.5% of the distinct pages ChatGPT pulled up for a question showed up in only one of the 20 runs.

It's a preprint, so it hasn't been peer reviewed, and every model it tested predates the GPT-5.6 and GPT-6 models OpenAI has since started rolling out in ChatGPT.

Sites are not brands, and coverage is not agreement

The paper's percentages measure recall: the share of sources cited in at least one of 20 ChatGPT app runs that also appeared in the external source sets, averaged across the 80 questions. They are not the percentage of two individual answers that matched.

A "site" here means a hostname, with subdomains counted separately. The combined Google and API source lists recovered 50.3% and 48.7% of cited hostnames, compared with 28.3% and 26.2% of exact cited URLs. A tool can therefore find the right site while missing the particular page ChatGPT selects.

The researchers also measured entities, including brands, extracted from source titles and snippets. They explicitly say those results do not support separate conclusions about brands. This paper does not establish how closely the API and app agree on which businesses to recommend.

There is another important limit: the paper counts supporting pages attached to selected references as cited, even if those supporting URLs are not displayed separately to the user. Its figures describe a broader reference-selection process, not only the clickable links a customer sees.

Why the API and the app give different answers

What the paper measured	ChatGPT app	External proxy setup
Questions and repetition	80 questions, 20 runs each on GPT-5.3 Instant and GPT-5.5 Instant	The same questions, 20 Responses API runs per model; up to five pages of Google organic results per question
Search queries	No exact match to the original question in observable GPT-5.3 and GPT-5.5 Instant searches	API web search enabled; Google searched the original question
Models and settings	GPT-5.3 Instant and GPT-5.5 Instant	gpt-5.3-chat-latest and gpt-5.5; no system prompt; reasoning effort set to none for gpt-5.5
Cited pages recovered by Google plus API candidates	App references are the comparison target	28.3% for GPT-5.3 Instant; 26.2% for GPT-5.5 Instant
Cited sites recovered by Google plus API candidates	Hostnames counted separately, including subdomains	50.3% for GPT-5.3 Instant; 48.7% for GPT-5.5 Instant

Source: Understanding ChatGPT Citations, Sections 2 and 4, Table 4. Coverage figures combine Google with the API's full candidate list, not just its final citations. Collections for each question were within a three-day window.

A few things are worth spelling out:

The question changes before the search happens. The paper found ChatGPT never searched with the exact wording of the question in its two newest tested settings. A check that assumes "question in, same search out" is measuring a different search from the one ChatGPT runs. We cover how that retrieval step works in our guide to how ChatGPT search works.

Model changes move everything. Search Engine Journal notes that around the GPT-5.5 change in May, Sistrix data showed day-to-day variation in German-language ChatGPT citations jumping to 47%, against a normal 1% to 2%. The paper's authors recommend re-checking your approach whenever the model behind ChatGPT changes, and noting which model produced each answer.

The app is noisy on its own. A separate study by Singapore digital agency First Page, using tracking data from 164,540 AI answers between October 9 and November 7, 2026, found ChatGPT re-cited only about half of the previous day's sites for the same question, and cited exactly the same sites as the day before on 2.7% of days. That study is upfront that its Perplexity numbers come from the Perplexity API, which "may not behave exactly like the Perplexity app." The same caveat applies to every engine.

What this means if you track AI visibility

None of this means API-based checks are useless. The paper found useful external signals, especially at the hostname level. The problem is reading more precision into them than they have. Here's how to use any AI visibility data more safely:

1. Treat brand mentions and page citations as two different metrics. Being named in an answer and having a page cited are different outcomes. This paper does not validate brand-recommendation agreement, so don't treat its site-level coverage as evidence that brand mentions match, or rebuild a page because one report says it lost a citation.
2. Track at the site level as well as the URL level. Combined, the stand-ins in the paper matched about twice as many cited sites as cited pages. "Our domain is cited" is a sturdier signal than "this URL is cited."
3. Run each question more than once. A single answer is one draw. The paper found substantial variation in retrieved pages across repeated app runs; that is not the same as measuring day-to-day citation loss. Use share over time, not a single screenshot. Our explainer on AI share of voice covers how to count that, and why one check is a snapshot covers the reasoning.
4. Write down the model and the date. When OpenAI changes the default model, compare before-and-after numbers separately instead of blending them.
5. Spot-check by hand. Pick five of your most important buyer questions and ask them in the ChatGPT app, logged out and logged in. If the brands match your report, the report is useful for trends. If they don't, ask why.

Questions to ask any AI visibility tool

The paper didn't test any commercial tool and cannot tell you how a specific tool collects its data. You can ask:

Question	Why it matters
Do answers come from the ChatGPT app or from the API?	The paper found that API source lists only partly recover app references
Which model and settings does each answer come from?	Model changes shift citations sharply
How many times is each question run?	One run is one sample, and the app varies run to run
Do you report brand mentions separately from cited pages?	Being named and supplying a cited source are different outcomes; this paper does not establish brand-level agreement
Are the questions unbranded?	Buyers ask "best X for Y", not your brand name
Can I see the raw answers?	You should be able to check the report against the text

The takeaway

The OpenAI API does not reproduce the ChatGPT app's sources. In this paper, even Google results plus the API's candidate list recovered only about a quarter of the cited pages, versus about half of the cited sites. Track sites and pages separately, repeat important questions, and record the collection method and model. If your goal is knowing whether ChatGPT recommends your business, inspect the actual answers too: these coverage figures do not validate brand recommendations.

To see where you stand today, Beket AI's free audit runs 25 unbranded buyer questions across ChatGPT, Perplexity and Google AI Overview and shows which businesses each engine names. Because a single run is a snapshot, Pro ($49/month) re-runs your questions daily, starting at 50 prompts across the same three engines, so you can follow the trend instead of one answer.

Sources: Jiang, Zhang, Xing, Chang and Xiao, "Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations" (arXiv preprint, October 11, 2026); Matt G. Southern, "OpenAI's API Only Partly Matched ChatGPT's Citations, Paper Finds", Search Engine Journal (October 18, 2026), including its Sistrix figure; First Page, "How Often AI Search Changes Its Sources" (November 7, 2026). Figures are as published by each source on the dates given.