Should You Block AI Crawlers On WordPress? Two Sites That Decided Say Yes, Allow Them.

Short answer: for most WordPress agency clients the sensible default is to allow AI crawlers, and to decide separately about training versus answering. If you are going to block ai crawlers wordpress sites need the distinction, because blocking the wrong agent removes you from the answers people are actively asking for.

I checked what real sites in this industry actually do rather than repeating the discourse, and checked again later to see whether any of them had changed their minds since.

What sites actually publish

blocking.*

Site	AI crawlers named	Directive	Reading
theplusaddons.com	29 user-agents	Allow: /	explicit opt-in for every named bot
wordpress.org	8+ named	Allow: /	GPTBot and ClaudeBot each explicitly allowed
protuno.com	none	silent	no rules, default applies: allowed
woocommerce.com	none	silent	same

Caveat: four sites, chosen because they are in our industry, not a survey of WordPress generally. What it shows reliably is that “allow” is a real, deliberate choice some sites make explicitly, not only the default nobody bothered to change.

That surprised me slightly. The conversation online is dominated by publishers blocking AI crawlers, and it is easy to absorb that as the default position. But publishers have a business model that WordPress product companies do not share.

One of ours, theplusaddons.com, names 29 user-agents and ends the block with Allow: /. That is a deliberate, considered opt-in, written out at real length, for every single bot somebody could think of.

Re-checked, plus two more sites, and our own practice matches our own advice

I re-read all four robots.txt files before expanding this post, and added two more of our own domains this post had not previously checked.

curl -s https://example.com/robots.txt \
  | grep -iE "GPTBot|ClaudeBot|Google-Extended|Content-Signal"

theplusaddons.com and wordpress.org both hold their original positions exactly: named agents, Allow: / on each. protuno.com still publishes no AI-crawler-specific rules. posimyth.com and nexterwp.com, not in the original table, extend the picture: posimyth.com matches protuno.com, silent, default-allow. nexterwp.com matches theplusaddons.com, both carrying the same named block naming GPTBot, ClaudeBot, ChatGPT-User, CCBot, Applebot-Extended and others, each allowed.

theplusaddons.com and nexterwp.com carry something the original table did not capture, a newer, separate directive beyond the traditional Allow/Disallow grammar:

Content-Signal: ai-train=yes, search=yes, ai-input=yes

That is the Content Signals proposal, layered on top of robots.txt rather than replacing it, answering a different question from crawl permission: not “can you fetch this” but “what may you do with it once fetched.” ai-train=yes on both of our sites carrying it is a direct match for this post’s own recommendation table, “software, SaaS, agency: allow everything,” which is worth having confirmed rather than assumed: our stated advice and our own sites’ actual configuration agree, on the two domains specific enough to declare it explicitly. wordpress.org, by contrast, allows the crawl through named Allow directives but declares no Content-Signal position at all, an intermediate stance worth naming on its own: permission to fetch, no explicit statement either way about training use.

Training and answering are different bots

Blocking ChatGPT-User and OAI-SearchBot removes you from the answers people are actively asking for, which is where referral traffic comes from.*

User-agent	Operator	What it feeds
GPTBot	OpenAI	model training
ChatGPT-User	OpenAI	fetching because a user asked about the page
OAI-SearchBot	OpenAI	search index for citation
ClaudeBot	Anthropic	model training
Claude-User	Anthropic	fetching on behalf of a user
Google-Extended	Google	Gemini training, does not affect Search ranking
PerplexityBot	Perplexity	search and citation
CCBot	Common Crawl	public dataset used by many models
Applebot-Extended	Apple	training, separate from Siri/Spotlight’s Applebot

Two details worth knowing before advising anybody.

Google-Extended controls Gemini training and does not affect Google Search ranking. People block it believing they are protecting their SEO, and they are not; they are opting out of one product while leaving Search untouched.

And robots.txt is a request, honoured voluntarily. Well-behaved crawlers respect it. Anything that does not care was never going to.

What the “29 user-agents” figure is actually made of

I counted the full, current list on theplusaddons.com rather than repeat the original figure without checking whether it still held, and it does, forty-three named User-agent blocks in total, but the number is worth breaking into what it actually contains rather than read as one undifferentiated pile of crawler names.

curl -s https://theplusaddons.com/robots.txt | grep '^User-agent:' | sort -u

Roughly a dozen of the forty-three, Nuclei, WikiDo, Riddler, Zoominfobot, Node/simplecrawler, dotbot/1.0 and similar, are blocked outright and have nothing to do with AI policy at all. They are security scanners and low-value scraping tools, the same category of generic bot-blocking any site runs regardless of its position on AI crawlers. The remaining roughly thirty are the actual AI and search block this post is about: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, PerplexityBot, Meta-ExternalAgent, MistralAI-User, Amazonbot, cohere-ai, DuckAssistBot, AI2Bot, Diffbot and more, every one of them carrying Allow: /.

That distinction matters for anyone reading a robots.txt file to size up a site’s actual AI policy rather than its general bot-management hygiene. A large total user-agent count on its own says nothing about whether a site welcomes or blocks AI crawlers specifically, since half of a long list can just as easily be routine scanner-blocking that would exist on the site regardless. The number worth reading is not “how many named agents,” it is “what does the Allow or Disallow say for the specific handful of names that matter,” GPTBot and ChatGPT-User foremost among them, and that smaller, specific reading is what actually answers the question this post is trying to help a reader decide.

Deciding, by what the client sells

took a position both said allow.*

Business	Sensible default	Reasoning
software, SaaS, agency, local service	allow everything	being cited is distribution, nobody buys instead of reading
publisher funded by advertising	allow answering, block training	the answer replaces the pageview that pays for it
paid research, courses, reports	block training	the content is the product
anything behind a login	not a robots.txt question	if it needs auth, robots.txt is the wrong control

The reasoning for that first row is simple. A plumber in Leeds does not lose a booking because an assistant summarised their opening hours. They gain one. An agency does not lose a client because a model read their case study.

The publisher case is genuinely different, and it deserves the honest answer rather than the same advice with more confidence. If the pageview is the product, an answer that replaces the pageview is a real cost.

Writing the rules to block AI crawlers on a WordPress site

# allow the crawlers that cite you, block the ones that only train
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

That is the split-decision version. For most clients the correct file is simpler: name nothing and allow everything by default, which is what protuno.com and woocommerce.com do.

Whatever you choose, verify it afterwards. A rule added to robots.txt is worth exactly as much as the rest of your robots.txt is accurate, and I have found sites blocking crawlers they meant to welcome.

The part I would not overthink

Nobody currently has a reliable, trustworthy measurement of what blocking actually costs or saves. Referral traffic from AI assistants is hard to attribute, training value is unmeasurable, and anyone quoting precise numbers is guessing.

So make it a genuine business decision, write the reasoning down clearly, and revisit it again in six months. Do not let it quietly turn into a technical project on its own.

What is worth the effort instead is making the content genuinely worth citing, which is a different piece of work entirely and the subject of getting cited by AI answer engines.

It also pairs with the file you publish for the same audience. An llms.txt is the opt-in half of this conversation, telling an agent what is worth reading once you have decided to let it in at all.

Protuno’s free audit reports the AI crawler policy from a domain’s robots.txt, so you can see across a portfolio which clients have taken a position and which have one by accident. Straight with you as on every post here: Iris, the SEO agent, is built and named but not live yet.

Read your clients’ robots.txt files this week. On the four sites checked for this post, two had no policy at all, silently defaulting to allow. Whether that ratio holds for your own clients is exactly what reading their files will tell you, and either way it is a decision worth having made on purpose.

The gap between “silent” and “decided against it”

Worth being precise about what a silent robots.txt actually means, since protuno.com and posimyth.com both fall into that category and it is easy to read silence as either carelessness or as an implicit policy, when it is neither. No GPTBot line, named or otherwise, means the general User-agent: * rules apply to it exactly as they apply to every other unnamed crawler. On both of our silent domains, that general rule is itself permissive, Allow: / with only a handful of specific paths disallowed, /api/, /dashboard, /admin, none of them AI-specific. The practical outcome is identical to theplusaddons.com‘s explicit thirty-name allow list: every AI crawler that respects robots.txt can fetch the site. The difference is entirely about whether that outcome was arrived at deliberately or inherited by default.

That distinction is worth making explicit to a client rather than leaving it implicit, because the two paths carry different risk profiles even when they produce the same crawl outcome today. An explicit list is a record of a decision someone can point to, defend, and revisit when Content-Signal or some future protocol adds a new distinction worth making. A silent robots.txt produces the same crawl behaviour by coincidence, for now, and will keep doing so right up until a future default changes, a new bot ships with a different fallback behaviour for unnamed crawlers, or someone adds a blanket rule for an unrelated reason and inadvertently catches AI crawlers in it. Writing the explicit version costs the same five minutes whether the decision is to allow everything or to split training from answering, and it is the version that survives someone else touching the file later without knowing what was originally intended.

Bot management is the same decision at a different layer

robots.txt states intent. Bot management enforces it, telling a good crawler from a bad one at the request level rather than trusting every visitor to read a text file and comply.