Is your website blocking AI crawlers?

Why this happens to people who never chose it

Almost nobody sits down and decides to block AI. It happens three ways. A security plugin or CDN ships with aggressive bot rules and treats new crawlers as scrapers. A developer or agency added a blanket block during the 2023–24 "AI is stealing our content" wave and never revisited it. Or your platform — some site builders and publishing tools — set a restrictive default you never saw. Two years on, the business wonders why AI never mentions them, and the answer has been sitting in a text file the whole time.

The two-minute check

• Open https://yourdomain.com/robots.txt in any browser. It's a plain text file — no tools needed.
• Scan for the AI user-agents listed below. You're looking for any Disallow: / sitting under them.
• Check for a blanket rule. A User-agent: * followed by Disallow: / blocks essentially everything, including assistants.
• Then check your CDN or firewall. This is the one people miss: Cloudflare, a WAF or a security plugin can block bots at the network level even when robots.txt says yes. If robots.txt looks fine but AI still can't fetch you, look here.

Who's who: the crawlers that matter

Not all bots do the same job. The important distinction is training crawlers (used to build models) versus retrieval/search crawlers (used to fetch pages when answering a live question). For being recommended, the retrieval ones matter most — they're how an assistant checks facts mid-answer.

The Google-Extended nuance worth understanding

Google-Extended isn't a crawler in its own right — it's a permission toggle. Blocking it doesn't remove you from Google Search (that's Googlebot), but it does restrict how your content may be used in Google's generative answers. So businesses sometimes block it to protect content from AI, then wonder why Gemini never names them. Both positions are defensible; what isn't defensible is holding them at the same time. Decide which you want.

What "allow" actually looks like

You don't need a clever configuration. For most businesses, the correct posture is simply: don't block the assistants you want to be recommended by. That means no Disallow: / under those user-agents, no blanket wildcard block, and a firewall that isn't quietly rejecting them. If your robots.txt has no AI-specific rules at all, that's usually fine — silence means allowed.

Two things worth adding while you're in there: a Sitemap: line pointing to your sitemap, so crawlers can find every page rather than guessing; and, if you like, an llms.txt file at your root summarising what your site is and where the key pages live. Be honest about that last one — llms.txt is a proposed convention, not a standard the major engines have committed to. It costs an hour and may help; it is not the thing that will make you visible. Don't let anyone sell it to you as a strategy.

What to do if you find a block

Remove the offending Disallow lines, save, and confirm the file updated by reloading the URL. Then give it time — assistants don't re-crawl on demand, and it can take days or weeks before the change shows up in answers. This is why it's worth checking first: everything else you invest in content, markup and citations is dead weight while the door is shut.

Frequently asked questions