> Every Protuno article: https://protuno.com/blog.md · Site map for AI agents: https://protuno.com/llms.txt

# Should You Block AI Crawlers On WordPress? Two Sites That Decided Say Yes, Allow Them

- Date: 14 September 2026
- Page: https://protuno.com/blog/wordpress-ai-crawler-policy

**Short answer:** for most WordPress agency clients the sensible default is to allow AI crawlers, and to decide separately about training versus answering. If you are going to block ai crawlers wordpress sites need the distinction, because blocking the wrong agent removes you from the answers people are actively asking for.

I checked what real sites in this industry actually do rather than repeating the discourse, and checked again later to see whether any of them had changed their minds since.

## What sites actually publish

![Table of AI crawler policies on four sites. siteb.example names 29 user-agents and allows all. wordpress.org explicitly allows GPTBot and ClaudeBot. sitea.example and woocommerce.com publish no AI crawler rules at all](https://cms.sitea.example/wp-content/uploads/2026/09/ai-crawler-policies-real-4.png)

*Two of these four take an explicit position, and both say allow. Nothing here is*

blocking.*

- **siteb.example**: AI crawlers named: 29 user-agents; Directive: `Allow: /`; Reading: explicit opt-in for every named bot
- **wordpress.org**: AI crawlers named: 8+ named; Directive: `Allow: /`; Reading: GPTBot and ClaudeBot each explicitly allowed
- **sitea.example**: AI crawlers named: none; Directive: silent; Reading: no rules, default applies: allowed
- **woocommerce.com**: AI crawlers named: none; Directive: silent; Reading: same

> **Caveat:** four sites, chosen because they are in our industry, not a survey of WordPress generally. What it shows reliably is that “allow” is a real, deliberate choice some sites make explicitly, not only the default nobody bothered to change.

That surprised me slightly. The conversation online is dominated by publishers blocking AI crawlers, and it is easy to absorb that as the default position. But publishers have a business model that WordPress product companies do not share.

One of ours, `siteb.example`, names 29 user-agents and ends the block with `Allow: /`. That is a deliberate, considered opt-in, written out at real length, for every single bot somebody could think of.

## Re-checked, plus two more sites, and our own practice matches our own advice

I re-read all four robots.txt files before expanding this post, and added two more of our own domains this post had not previously checked.

```
curl -s https://example.com/robots.txt 
  | grep -iE "GPTBot|ClaudeBot|Google-Extended|Content-Signal"
```

`siteb.example` and `wordpress.org` both hold their original positions exactly: named agents, `Allow: /` on each. `sitea.example` still publishes no AI-crawler-specific rules. `sitec.example` and `sited.example`, not in the original table, extend the picture: `sitec.example` matches `sitea.example`, silent, default-allow. `sited.example` matches `siteb.example`, both carrying the same named block naming `GPTBot`, `ClaudeBot`, `ChatGPT-User`, `CCBot`, `Applebot-Extended` and others, each allowed.

`siteb.example` and `sited.example` carry something the original table did not capture, a newer, separate directive beyond the traditional `Allow`/`Disallow` grammar:

```
Content-Signal: ai-train=yes, search=yes, ai-input=yes
```

That is the [Content Signals](https://contentsignals.org) proposal, layered on top of robots.txt rather than replacing it, answering a different question from crawl permission: not “can you fetch this” but “what may you do with it once fetched.” `ai-train=yes` on both of our sites carrying it is a direct match for this post’s own recommendation table, “software, SaaS, agency: allow everything,” which is worth having confirmed rather than assumed: our stated advice and our own sites’ actual configuration agree, on the two domains specific enough to declare it explicitly. `wordpress.org`, by contrast, allows the crawl through named `Allow` directives but declares no Content-Signal position at all, an intermediate stance worth naming on its own: permission to fetch, no explicit statement either way about training use.

## Training and answering are different bots

![Table of nine AI crawler user-agents with their operators and purpose. GPTBot and ClaudeBot feed model training. ChatGPT-User and Claude-User fetch on behalf of a user. OAI-SearchBot and PerplexityBot feed search and citation. Google-Extended feeds Gemini training and does not affect Search ranking](https://cms.sitea.example/wp-content/uploads/2026/09/ai-crawler-agents-4.png)

*This distinction is the whole decision. Blocking `GPTBot` removes you from training data.*

Blocking `ChatGPT-User` and `OAI-SearchBot` removes you from the answers people are actively asking for, which is where referral traffic comes from.*

- **`GPTBot`**: Operator: OpenAI; What it feeds: model training
- **`ChatGPT-User`**: Operator: OpenAI; What it feeds: fetching because a user asked about the page
- **`OAI-SearchBot`**: Operator: OpenAI; What it feeds: search index for citation
- **`ClaudeBot`**: Operator: Anthropic; What it feeds: model training
- **`Claude-User`**: Operator: Anthropic; What it feeds: fetching on behalf of a user
- **`Google-Extended`**: Operator: Google; What it feeds: Gemini training, does not affect Search ranking
- **`PerplexityBot`**: Operator: Perplexity; What it feeds: search and citation
- **`CCBot`**: Operator: Common Crawl; What it feeds: public dataset used by many models
- **`Applebot-Extended`**: Operator: Apple; What it feeds: training, separate from Siri/Spotlight’s Applebot

Two details worth knowing before advising anybody.

`Google-Extended` controls Gemini training and **does not affect Google Search ranking**. People block it believing they are protecting their SEO, and they are not; they are opting out of one product while leaving Search untouched.

And robots.txt is a request, honoured voluntarily. Well-behaved crawlers respect it. Anything that does not care was never going to.

## What the 29 user-agents figure is actually made of

I counted the full, current list on `siteb.example` rather than repeat the original figure without checking whether it still held, and it does, forty-three named `User-agent` blocks in total, but the number is worth breaking into what it actually contains rather than read as one undifferentiated pile of crawler names.

```
curl -s https://siteb.example/robots.txt | grep '^User-agent:' | sort -u
```

Roughly a dozen of the forty-three, `Nuclei`, `WikiDo`, `Riddler`, `Zoominfobot`, `Node/simplecrawler`, `dotbot/1.0` and similar, are blocked outright and have nothing to do with AI policy at all. They are security scanners and low-value scraping tools, the same category of generic bot-blocking any site runs regardless of its position on AI crawlers. The remaining roughly thirty are the actual AI and search block this post is about: `GPTBot`, `ChatGPT-User`, `OAI-SearchBot`, `ClaudeBot`, `anthropic-ai`, `Claude-Web`, `Google-Extended`, `PerplexityBot`, `Meta-ExternalAgent`, `MistralAI-User`, `Amazonbot`, `cohere-ai`, `DuckAssistBot`, `AI2Bot`, `Diffbot` and more, every one of them carrying `Allow: /`.

That distinction matters for anyone reading a robots.txt file to size up a site’s actual AI policy rather than its general bot-management hygiene. A large total user-agent count on its own says nothing about whether a site welcomes or blocks AI crawlers specifically, since half of a long list can just as easily be routine scanner-blocking that would exist on the site regardless. The number worth reading is not “how many named agents,” it is “what does the `Allow` or `Disallow` say for the specific handful of names that matter,” `GPTBot` and `ChatGPT-User` foremost among them, and that smaller, specific reading is what actually answers the question this post is trying to help a reader decide.

## Deciding, by what the client sells

![Table of business types and sensible defaults. Software, SaaS, agencies and local services should allow everything. Ad-funded publishers should allow answering and block training. Paid research and courses should block training. Anything behind a login is not a robots.txt question](https://cms.sitea.example/wp-content/uploads/2026/09/ai-crawler-decision-4.png)

*The first row covers most WordPress agency clients, which is why the two sites above that*

took a position both said allow.*

- **software, SaaS, agency, local service**: Sensible default: allow everything; Reasoning: being cited is distribution, nobody buys instead of reading
- **publisher funded by advertising**: Sensible default: allow answering, block training; Reasoning: the answer replaces the pageview that pays for it
- **paid research, courses, reports**: Sensible default: block training; Reasoning: the content is the product
- **anything behind a login**: Sensible default: not a robots.txt question; Reasoning: if it needs auth, robots.txt is the wrong control

The reasoning for that first row is simple. A plumber in Leeds does not lose a booking because an assistant summarised their opening hours. They gain one. An agency does not lose a client because a model read their case study.

The publisher case is genuinely different, and it deserves the honest answer rather than the same advice with more confidence. If the pageview is the product, an answer that replaces the pageview is a real cost.

## Writing the rules to block AI crawlers on a WordPress site

```
# allow the crawlers that cite you, block the ones that only train
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /
```

That is the split-decision version. For most clients the correct file is simpler: name nothing and allow everything by default, which is what `sitea.example` and `woocommerce.com` do.

Whatever you choose, verify it afterwards. A rule added to robots.txt is worth exactly as much as [the rest of your robots.txt is accurate](https://protuno.com/blog/wordpress-robots-txt-blocking), and I have found sites blocking crawlers they meant to welcome.

## The part I would not overthink

Nobody currently has a reliable, trustworthy measurement of what blocking actually costs or saves. Referral traffic from AI assistants is hard to attribute, training value is unmeasurable, and anyone quoting precise numbers is guessing.

So make it a genuine business decision, write the reasoning down clearly, and revisit it again in six months. Do not let it quietly turn into a technical project on its own.

What is worth the effort instead is making the content genuinely worth citing, which is a different piece of work entirely and the subject of [getting cited by AI answer engines](https://protuno.com/blog/wordpress-ai-citations).

It also pairs with the file you publish for the same audience. An [llms.txt is the opt-in half of this conversation](https://protuno.com/blog/llms-txt-wordpress), telling an agent what is worth reading once you have decided to let it in at all.

Protuno’s free audit reports the AI crawler policy from a domain’s robots.txt, so you can see across a portfolio which clients have taken a position and which have one by accident. **Iris is live now.**

Read your clients’ robots.txt files this week. On the four sites checked for this post, two had no policy at all, silently defaulting to allow. Whether that ratio holds for your own clients is exactly what reading their files will tell you, and either way it is a decision worth having made on purpose.

## The gap between silent and decided against it

Worth being precise about what a silent robots.txt actually means, since `sitea.example` and `sitec.example` both fall into that category and it is easy to read silence as either carelessness or as an implicit policy, when it is neither. No `GPTBot` line, named or otherwise, means the general `User-agent: *` rules apply to it exactly as they apply to every other unnamed crawler. On both of our silent domains, that general rule is itself permissive, `Allow: /` with only a handful of specific paths disallowed, `/api/`, `/dashboard`, `/admin`, none of them AI-specific. The practical outcome is identical to `siteb.example`‘s explicit thirty-name allow list: every AI crawler that respects robots.txt can fetch the site. The difference is entirely about whether that outcome was arrived at deliberately or inherited by default.

That distinction is worth making explicit to a client rather than leaving it implicit, because the two paths carry different risk profiles even when they produce the same crawl outcome today. An explicit list is a record of a decision someone can point to, defend, and revisit when Content-Signal or some future protocol adds a new distinction worth making. A silent robots.txt produces the same crawl behaviour by coincidence, for now, and will keep doing so right up until a future default changes, a new bot ships with a different fallback behaviour for unnamed crawlers, or someone adds a blanket rule for an unrelated reason and inadvertently catches AI crawlers in it. Writing the explicit version costs the same five minutes whether the decision is to allow everything or to split training from answering, and it is the version that survives someone else touching the file later without knowing what was originally intended.

## Bot management is the same decision at a different layer

*Good Bots, Bad Bots: How to Win with Bot Management*

`robots.txt` states intent. Bot management enforces it, telling a good crawler from a bad one at the request level rather than trusting every visitor to read a text file and comply.
