Technical AI Crawlers 4 Min Read

Does robots.txt Block ChatGPT from Your Website? (Here's What's Actually Happening)

You set up your website, published great content, and waited for AI visitors to arrive. But when you search for your business on ChatGPT, it acts like your site doesn't exist. Before you rewrite your entire homepage, check one simple file: your robots.txt.

THE SHORT ANSWER (WITHOUT THE JARGON)

Yes, a single line in your robots.txt file can quietly shut out ChatGPT. OpenAI's web crawler identifies itself as GPTBot. If your site or web host automatically added Disallow: / for GPTBot, ChatGPT is strictly forbidden from visiting or quoting your pages.

Check Your Bot Access Rules in 5 Seconds

Unsure if Cloudflare or your WordPress host is blocking AI bots? Type your domain below to test your rules live.

Why Web Hosts Block AI Crawlers Without Telling You

Here is the frustrating part: you might not have blocked ChatGPT yourself. Over the past two years, several popular web hosts and security plugins (like Cloudflare, WP Engine, and SiteGround) started enabling default "Block AI Crawlers" toggles to save server bandwidth.

While that protects your server from aggressive web scrapers, it also hides your site from modern AI search engines like ChatGPT Search and Perplexity.

How to Fix Your robots.txt Rules

Open your browser and visit https://yourwebsite.com/robots.txt. Look for any section containing User-agent: GPTBot or User-agent: PerplexityBot.

If you see Disallow: / under those bots, replace it with this clean, friendly configuration:

# Allow AI search engines to index public pages
User-agent: GPTBot
Allow: /
Disallow: /admin/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/

Pro Tip: Give ChatGPT 24-48 hours after updating your file to re-crawl your domain and index your latest changes.