6 min read
Should you block GPTBot in robots.txt?
Blocking GPTBot stops training use, not ChatGPT search visibility. See what each OpenAI, Google, Anthropic and Perplexity crawler does and copy the rules.
Block GPTBot only if you don't want your content used to train OpenAI's models. Blocking it does not remove you from ChatGPT search answers, because a separate crawler, OAI-SearchBot, controls that. For most businesses that want to be named by AI assistants, the usual setup is to disallow GPTBot if training use bothers you and to keep search crawlers allowed.
What does blocking GPTBot actually do?
Blocking GPTBot tells OpenAI not to use your content for training its generative AI foundation models. OpenAI says each of its robots.txt settings is independent of the others.
OpenAI's own documentation describes the combination many sites want: allow OAI-SearchBot so you appear in search results, and disallow GPTBot so crawled content isn't used for training. Sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links. Details are in OpenAI's crawler overview.
Which OpenAI crawlers matter?
Two OpenAI user agents are controlled through robots.txt, and a third works differently.
- GPTBot: collects content that may be used for training. Disallowing it signals that your content shouldn't be used for training.
- OAI-SearchBot: decides whether your pages show up in ChatGPT search answers. Disallowing it removes you from those answers.
- ChatGPT-User: fetches pages when a user asks for them. OpenAI says these actions are user-initiated, so robots.txt rules may not apply, and it isn't used to decide Search inclusion.
OpenAI also says search systems can take about 24 hours to adjust after you change robots.txt.
What about Google, Anthropic and Perplexity?
Each company runs its own crawlers, and each publishes its own list of user agents and what they do. Tokens change, and a token that controls training at one company may control something different at another, so check each company's crawler documentation before you deploy rules.
Perplexity. Perplexity publishes two user agents in its crawler documentation:
- PerplexityBot: surfaces and links websites in Perplexity search results. Perplexity states that this bot isn't used to gather content for AI foundation models. Blocking it keeps you out of those search results.
- Perplexity-User: fetches a page when a user asks for it. Perplexity states that this fetcher runs because a user asked for the page, so it generally skips robots.txt rules and blocking it may do nothing.
Google. Google runs more than one user agent, and they do different jobs. Googlebot is the crawler behind Google Search. Blocking it takes your pages out of the crawling that Search relies on, so most sites leave it allowed.
Google-Extended is a standalone product token, which means a name you use in robots.txt that is not a separate crawler. Publishers can use it to decide whether the content Google crawls from their sites may help train future Gemini models. It has no effect on inclusion in Google Search and is not a ranking signal. That is why the example below blocks Google-Extended and allows Googlebot. Both are described on Google's common crawlers page. See Google's robots.txt introduction for how robots.txt rules work.
Anthropic. Anthropic runs three user agents: ClaudeBot, Claude-SearchBot and Claude-User. Anthropic says restricting ClaudeBot tells it to leave your site's future content out of its AI training data. Anthropic says that blocking Claude-SearchBot stops it from indexing your content for search, so your site may appear less often, and less accurately, in Claude's search results. Claude-User fetches pages when a person asks Claude about them. Anthropic also says blocking its IP addresses may not work as an opt-out, because it stops Anthropic from reading your robots.txt file. All of this is on Anthropic's crawler help page.
Blocking one company's bot doesn't block any other company's bot. You need a rule group for each user agent you want to address.
Does robots.txt really keep crawlers out?
Robots.txt works for responsible bots, but compliance is voluntary. It is a deterrent, not a prevention method, and nobody centrally monitors whether crawlers follow it. If you need a hard block, add rules at the server or firewall level.
The reverse problem also exists. A firewall, CDN, or bot-protection service may apply separate restrictions, so a robots.txt that allows a crawler doesn't guarantee the crawler can reach your pages. OpenAI notes this in its guidance for allowing its crawlers.
Blocking GPTBot also doesn't undo past crawling. Content already collected stays collected.
How to set your rules, step by step
- Decide what you want. Pick one: allow everything, block training only, or block everything. This sets the rest of the rules.
- Open the robots.txt file at the root of your domain, such as yourdomain.com/robots.txt. Rules only work from that location.
- Add a group for each training crawler you want to block. This stops training use without touching search.
- Add explicit Allow groups for search crawlers like OAI-SearchBot. This keeps you eligible for ChatGPT search answers.
- Remove blanket rules that catch everything. A wildcard disallow for all bots, or a rule that blocks every agent with "GPT" in its name, takes OAI-SearchBot out along with GPTBot, and your site drops out of ChatGPT's search-based answers.
- Save, deploy, and wait about a day. OpenAI says its search systems can take around 24 hours to adjust.
- Check your firewall and CDN settings. This confirms the crawlers you allowed can reach your pages.
Example robots.txt rules
To block training but stay visible in ChatGPT search and Google Search:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
To block training on only part of your site, disallow paths instead of the whole site:
User-agent: GPTBot
Disallow: /members/
Disallow: /research/
Allow: /
To stay fully open, leave these agents out of robots.txt or give each an Allow: / rule.
Which choice fits a business that wants AI visibility?
If buyers ask AI assistants which vendor to pick, being crawled by search bots is how you get named. Block training crawlers if you object to training use, and keep search crawlers open. A full block removes you from the answers you want to appear in.
Once your rules are set, it helps to know whether assistants name you. For how citations work, see how to get cited by AI and AI mention vs AI citation.
Frequently asked questions
Will blocking GPTBot hide my site from ChatGPT?
Blocking GPTBot alone does not hide your site from ChatGPT search, because OAI-SearchBot controls search inclusion separately. Your site only drops out if you disallow OAI-SearchBot or use a wildcard that catches it.
Do I need a rule for ChatGPT-User?
A rule for ChatGPT-User may not do anything, since OpenAI says robots.txt rules may not apply to actions a user starts. Use OAI-SearchBot to manage search opt-outs.
Is llms.txt a replacement for robots.txt?
These files do different jobs. Robots.txt controls whether a crawler visits, while llms.txt is meant to guide how an AI understands and uses your content.
Check whether AI names you
After you set your rules, find out what assistants actually say about your business. See whether AI names you with our free check. It takes about two minutes, no sign-up needed.
See what AI says about you
Start free. Pick a plan when you're ready for tracking and content.