ChatGPT and robots.txt: what 50 French sites reveal (and the checklist to become citable)
Your next creation starts with a sentence. — free credits on sign-up Bring my idea to life
For the past year, the most frequent SEO question is no longer "how do I outrank a competitor on Google" but "how do I get ChatGPT to cite my site". The answer rarely starts with content: it starts with a text file few people read, the robots.txt. A guest article published on Webmarketing & co'm reports reading 50 robots.txt files from French sites in early November 2026, spanning press, tech media, e-commerce, software publishers and travel. Here is what those observations suggest, and a checklist to act on — with no guarantee of citation.
What a robots.txt actually controls
A robots.txt does not "rank" a site: it allows or forbids crawlers from browsing it. And AI crawlers do not all serve the same purpose. According to OpenAI documentation cited in the article, GPTBot is used for model training, OAI-SearchBot for ChatGPT search and ChatGPT-User to fetch a page for a question. These settings are independent: you can allow OAI-SearchBot and block GPTBot, separating citation from training. On Perplexity's side, PerplexityBot is used to index and link sites, while Perplexity-User generally ignores robots.txt because the request comes from a human, again per the documentation reported.
Practical consequence: "blocking AI" has no single meaning. The useful question is: which crawler, for which use, and do I want to be cited or only protected?
What the sample shows
Of the 50 sites, 43 files were readable; seven did not respond, six of them with a 403 refusal. Across those 43 files, the author reports that 19 restrict at least one training crawler, but only 10 restrict a citation crawler — nine of them press titles. In other words, blocking training is more common than blocking citation.
One detail stands out: seven of the ten sites restricting a citation crawler reopen commercial or affiliate sections via Allow rules. Libération, France TV Info and Le Monde close without exception, Le Monde only on Anthropic's search crawler. Outside press, no site in the sample restricts a citation crawler: zero in e-commerce, software and travel. Twenty of the 43 sites have no named rule targeting AI, leaving them open by default.
Another finding: CCBot, the Common Crawl crawler, is the most blocked (16 of 43 sites), a rule often inherited from 2023 debates. And eleven of 50 sites publish an llms.txt file, mostly software publishers and travel players.
Caution: this is a small, non-representative sample of French sites, and the observations are reported by the author, not independent tests. The author also does not settle whether publishers keeping commercial sections open will still do so after signing licensing deals.
Why robots.txt comes before content
You can write the best article in the world: if the citation crawler is blocked, it will not be read to answer. Conversely, a site open by default is not cited for all that — openness is a condition, not a result. That is the central nuance: the checklist below aims to stop you blocking yourself, not to guarantee a place in answers.
Practical checklist
1. Audit named rules
List the User-agent entries in your robots.txt and map each to a use: training (GPTBot, CCBot), citation (OAI-SearchBot, PerplexityBot), on-demand fetch (ChatGPT-User). Check Allow and Disallow rules, including those inherited from older debates.
2. Decide by use, not by reflex
Want to be cited? Then the citation crawler must be able to browse the pages you want cited. Want to limit training? That is a separate choice, technically separable. Document the decision so a future change does not contradict it.
3. Check tags that block display
According to Google, nosnippet, data-nosnippet, max-snippet and noindex tags can prevent display as a supporting link in AI Overviews. Content open in robots.txt but closed by tag remains invisible.
4. Improve content citability
The Princeton GEO study (November 2023) reports that citing sources, adding experts and adding figures improves visibility by 30–40% and 15–30% depending on the metric. The authors note effectiveness varies strongly by domain: treat it as a lead, not a recipe.
5. Do not rely on Search Console to measure
Search Console does not break out traffic from AI features: it is counted in overall web traffic. Measuring visibility in AI answers remains manual, with no reference tool.
6. Beware miracle files
Google states that no AI-specific text file or markup is required for its AI features. llms.txt is an emerging convention, not a guaranteed standard.
Quick comparison of the crawlers cited
- GPTBot (OpenAI): training. Blocked by many sites in the sample.
- OAI-SearchBot (OpenAI): ChatGPT search. Rarely blocked outside press.
- ChatGPT-User (OpenAI): fetching a page for a question.
- PerplexityBot: indexing and links. Perplexity-User: generally ignores robots.txt.
- CCBot (Common Crawl): the most blocked (16/43), often by inheritance.
Key takeaways
Visibility in ChatGPT starts with a readable technical decision: which crawlers do you allow, for which use. The sample's observations show many sites block training without blocking citation, and that press is the most restrictive. But openness guarantees nothing: citation also depends on content quality and citability, and measurement remains imperfect. Treat robots.txt as a prerequisite, not a promise.
This article is published by Roboto, a platform for generating texts, images, videos and voiceovers with AI. It is not based on any testing performed by Roboto: the observations reported come from the cited source article.
Source
Visibilité dans ChatGPT : j’ai lu le robots.txt de 50 sites français