Easy Design Agency Get the free audit
  1. Home
  2. Short answers
  3. Does blocking AI crawlers hurt a local business?

Short answer: AI answers

Does blocking AI crawlers hurt a local business?

  • By Sasha, Easy Design Agency
  • Updated November 5, 2026
  • 2 min read

For a local service business, yes. The AI assistants that homeowners ask for a roofer or a plumber fetch live pages to build their answer; if your robots.txt or a hosting setting blocks their crawlers, they cannot read yours and they describe the companies they can read.

The block is usually not a decision anyone made: a security plugin, a managed firewall or a host's "block AI bots" toggle turned it on. Publishers with paywalled writing have a real case for blocking. A plumber does not.

Which crawlers matter

For being found: OpenAI's OAI-SearchBot and ChatGPT-User, Anthropic's Claude-SearchBot and Claude-User, Perplexity's PerplexityBot and Perplexity-User.1,2,3 Google's AI Overviews use the ordinary Googlebot, which almost nobody blocks; the Google-Extended token only controls Gemini training and has no effect on whether you appear in Overviews.4 Training crawlers (GPTBot, ClaudeBot, CCBot) matter less for a local business, but blocking them buys nothing either; the pages are a description of your services, not a novel.

Where the block comes from

  • A security or SEO plugin with an "AI bots" switch, often on by default after an update.
  • A managed robots.txt from the host or a CDN that prepends its own rules to yours. Cloudflare, for one, offers a managed robots.txt and an "AI Crawl Control" that can block by default; ours had to be switched off.
  • A firewall rule that blocks by user agent and returns an error page, which a crawler reads as "nothing here".
  • Someone's good intention in 2023, never revisited.

How to check and fix it

  1. Open yourdomain.com/robots.txt on your phone. Look for User-agent: lines naming any of the crawlers above followed by Disallow: /.
  2. Check the host or CDN dashboard for an AI-bot or bot-fight setting and turn it off for the named crawlers.
  3. Add the crawlers to your own robots.txt with Allow: / by name, so a rule added elsewhere later is not read as a block. Ours is at /robots.txt.
  4. Request the free audit; its answer-engine check lists which crawlers your site currently blocks.

The AI answers guide covers what the assistants do once they can read you.

Will allowing crawlers slow my site or cost bandwidth?

Not noticeably for a small site; these crawlers fetch a few pages occasionally. Heavy scraping comes from unnamed bots that ignore robots.txt anyway.

What about my photos being used to train models?

Allow the search and user-fetch crawlers and, if it matters to you, block only the training ones (GPTBot, ClaudeBot, CCBot, Google-Extended). You keep the visibility and make the point.

Sources

  1. OpenAI: Overview of OpenAI crawlers.
  2. Anthropic: Does Anthropic crawl data from the web?
  3. Perplexity: Perplexity crawlers.
  4. Google Search Central: Overview of Google crawlers and fetchers, Google-Extended.

Read next

Guide: AI answers

How AI answers pick contractors

Ask ChatGPT, Google's AI or Perplexity for a roofer and you get three names. Where that list comes from, what gets a page quoted, which crawlers to allow.

7 min readShort answer: AI answers

What is llms.txt?

A plain-text file at yourdomain.com/llms.txt that tells AI assistants what the business is and which pages to read. Ten minutes to add; not a ranking signal.

2 min readShort answer: AI answers

Google's AI Overviews for contractors

When Google answers a local search with its own summary, the companies it names come from the map and web results you already compete for. What changes.

2 min read

See this on your own site.

The free audit measures your site on a real phone: load time, the first screen, the call button, what Google's AI says about you next to the competitors. Written, back within one business day. No call.

Get the free audit