The instruction to block gptbot robots txt gets passed around as though it were obviously correct, and for some publishers it is. For most businesses it is a decision with a real cost on both sides, and the cost is easy to trigger by accident because the crawlers doing very different jobs have confusingly similar names.
Blocking the wrong one removes you from AI answers entirely. Blocking one particular crawler removes you from Google.
Which crawler does what
| Crawler | Run by | What it does | Blocking it means |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for training future models | Not used in training. No effect on citation |
| OAI-SearchBot | OpenAI | Fetches pages live for ChatGPT search | You cannot be cited in ChatGPT answers |
| ChatGPT-User | OpenAI | Fetches a page when a user asks about it directly | ChatGPT cannot open links to your site |
| ClaudeBot | Anthropic | Training collection | Not used in training |
| PerplexityBot | Perplexity | Indexes for live answers | You cannot appear in Perplexity results |
| Google-Extended | Controls use for Gemini training only | Not used for Gemini. Search unaffected | |
| Googlebot | Search index, and AI Overviews | You vanish from Google search | |
| Bingbot | Microsoft | Bing index, and Copilot | You vanish from Bing and Copilot |
There is no way to appear in Google search while opting out of AI Overviews. They are served from the same index. Blocking Googlebot to avoid AI Overviews removes you from Google completely. Businesses do this, and the traffic loss is immediate and total.
The trade-off, stated plainly
Allowing crawlers means your work may be used to train systems that can answer questions without sending anyone to your site. That is a genuine grievance and it is the reason the debate exists.
Blocking retrieval crawlers means that when somebody asks ChatGPT or Perplexity to recommend a service like yours, your business is not among the options. Your competitors who allowed access are.
Does robots.txt even work
Partly, and it is worth being clear about the limits.
robots.txt is a voluntary standard from 1994. It has no enforcement. It works because established companies choose to honour it, and OpenAI, Anthropic, Google and Perplexity all state that their crawlers do.
What it does not do is stop anyone determined. Scrapers built to harvest content ignore it, and some use browser-like user agents so they are indistinguishable from visitors. If your concern is a bad actor rather than a named company, robots.txt is not the tool.
It also cannot undo anything. Content already collected is already collected. Blocking today affects future crawling only.
How to write the rules
Your robots.txt sits at yourdomain.com/robots.txt. In WordPress, Rank Math and Yoast both let you edit it without touching files.
To block training while staying citable in AI answers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /To block everything AI-related, including citation:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /To block only part of a site, which is often the sensible middle path:
User-agent: GPTBot
Disallow: /research/
Disallow: /courses/
Allow: /Open yourdomain.com/robots.txt and read what is there now. Plugins, hosts and security tools all add rules, and sites frequently carry blocks nobody remembers adding. Confirm the file loads correctly after every change, because a syntax error in robots.txt can be read far more broadly than intended.
Blocking at server level
If a crawler is ignoring robots.txt, or you want a firmer boundary, blocking happens above WordPress.
Cloudflare offers a one-click AI Scrapers and Crawlers setting under Security, which blocks known AI bots at the edge before they reach your server. It is the simplest option for most sites and it costs nothing on the free plan.
Beyond that, user-agent rules in .htaccess or a firewall rule at the host level will work. Both are more brittle: user agents change, and an over-broad rule can block legitimate services. Test carefully, and check your search traffic afterwards.
What we would suggest, by situation
- A normal business site, marketing contentAllow everything. You want to be found, and being absent from AI answers costs more than training use does.
- A site with paid or original researchBlock training crawlers on the paid sections, allow retrieval everywhere. You stay citable without giving away the asset.
- A publisher earning from page viewsThe genuinely hard case. Blocking retrieval protects the click and reduces reach. Many are negotiating licensing instead.
- Anything private or client-ownedrobots.txt is the wrong tool. Put it behind a login.
- An ecommerce siteAllow everything. Product visibility in AI shopping answers is becoming a real channel.
Why does this content exist? If it was published to bring in customers, blocking the systems people ask for recommendations works against the reason you wrote it. If it is the product itself, protect it properly rather than with a text file.
What not to do
- Do not block Googlebot to avoid AI Overviews. You will lose Google search entirely.
- Do not copy a robots.txt from another site. You will inherit rules that do not fit yours.
- Do not block everything as a precaution. Decide deliberately, per crawler.
- Do not rely on robots.txt for anything confidential. It is a request, and the file itself is public.
- Do not forget to check it after a migration. Staging robots.txt files reaching production is one of the most common causes of a site disappearing from search.
A five minute audit
Open your robots.txt now and read it. Note which AI crawlers appear, whether Googlebot is mentioned at all, and whether anything is blocked that you did not intend. Then decide, once, what you actually want and write it down alongside the file so the next person does not undo it.
Most sites we look at have never had this decision made deliberately. The rules present are whatever a plugin added or an old developer left behind. Reviewing it is a fifteen minute job that sits naturally within technical SEO work, and it is worth doing before it becomes the reason a site quietly stopped appearing anywhere. If your robots.txt is generated by a plugin or a caching layer rather than a real file, our development team can sort out which one is actually in charge.
Parshant Hadala