Tools
Browser, Terminal (curl), Claude or ChatGPT
Price
Free
Time
2 min to test, 15 min to fix
Level
Beginner

You type your company name into ChatGPT or Perplexity. The answer is vague, out of date, or mentions a competitor. There can be several reasons, but the first one to rule out is simple: your website may be blocking AI crawlers, sometimes for years, without anyone having decided it.

This tutorial is for any small business owner or freelancer with a website. You do not need to be a developer: you read a text file, run one command and ask an AI assistant to do the diagnosis.

What you need to know about robots.txt

The robots.txt file sits at the root of your site (for example yoursite.com/robots.txt). It tells crawlers which pages they may visit. Each AI company runs several crawlers with different jobs. Here are the official names, as published by OpenAI, Anthropic, Perplexity and Google:

  • OpenAI: OAI-SearchBot surfaces websites in ChatGPT search, GPTBot collects content that may be used to train models, ChatGPT-User visits a page when a user asks for it.
  • Anthropic (Claude): Claude-SearchBot is for search, ClaudeBot for training, Claude-User for pages requested by a Claude user.
  • Perplexity: PerplexityBot shows and links sites in its results (Perplexity says it does not train models), Perplexity-User visits a page on a user's request.
  • Google: Googlebot for Google Search. Google-Extended is not a crawler but a token that controls whether your content is used to train Gemini. Google states it has no effect on your inclusion or ranking in Search.

Remember the difference: blocking a training crawler does not remove you from answers. Blocking a search crawler does.

The test, step by step

  1. Read your robots.txt fileOpen yoursite.com/robots.txt in your browser, or run this command in the Terminal (Mac or Linux). Replace the address with yours.
    Terminal
    curl -s https://www.yoursite.com/robots.txt
  2. Spot the blocking linesLook for Disallow: / under User-agent: * (everyone is blocked) or under the name of one of the crawlers above. An error page instead of the file is fine: with no robots.txt, nothing is forbidden.
  3. Test how your server answers each crawlerSome hosts and firewalls block AI crawlers before robots.txt even matters. This command requests your home page under each crawler's name and prints the code it gets back: 200 means "page served", 403 means "refused".
    Terminal
    for bot in OAI-SearchBot GPTBot Claude-SearchBot ClaudeBot PerplexityBot Googlebot; do
  echo "$bot : $(curl -s -o /dev/null -w '%{http_code}' -A "$bot" https://www.yoursite.com/)"
done
  4. Let the AI read the resultPaste your robots.txt and the codes into Claude or ChatGPT with the prompt below. Nothing in it is confidential: robots.txt is a public file.
Prompt
Help me check whether AI crawlers can read my website.

Here is my robots.txt file:
<robots>
[paste the content here]
</robots>

Here are the HTTP codes my home page returned for each crawler:
<codes>
[paste the command output here]
</codes>

1. Make a table: crawler, company, role (search, training, on-demand visit), allowed or blocked, the line of the file that decides it.
2. Point out syntax errors and rules that contradict each other.
3. If a code is 403 while robots.txt allows the crawler, explain that the block probably comes from the host or firewall, and tell me where to look.
4. Suggest a corrected version that allows search crawlers and lets me choose for training crawlers.

Base your answer only on the file and codes provided. If you do not know a crawler name, say so instead of guessing.

Fix the file

If you want to appear in AI answers while refusing training, here is a template. Lines starting with # are comments. Several User-agent lines in a row share the same rule.

robots.txt
# Classic search and AI search: allowed
User-agent: Googlebot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Model training: refused here, your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Every other crawler
User-agent: *
Allow: /

Sitemap: https://www.yoursite.com/sitemap.xml

On WordPress, Wix or Shopify, the file is often edited from the SEO settings rather than the command line. If your site runs through Cloudflare, also check the AI crawler settings under Security Settings: Cloudflare changed its defaults for new domains on 15 September 2026.

Good to know

  • robots.txt is a request, not a lock. Google reminds us that it cannot enforce crawler behavior and is not a way to hide a page. Protect a private page with a password.
  • Visits made on a user's request follow other rules: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores those rules.
  • OpenAI says it can take about 24 hours for a robots.txt change to be reflected in search.
  • The curl test only imitates the crawler's name. Some firewalls recognize real crawlers by IP address and may treat an imitator differently: a 403 is a clue, not proof.
  • Being readable does not guarantee being cited. It is the starting condition; after that, the content of your pages makes the difference.

Before you edit. Keep a copy of your current robots.txt. A Disallow: / placed under User-agent: * by mistake removes your whole site from search engines. Re-read the file after every change.

Sources

Want a site that Google and AI assistants can read from day one? I build that into every website I deliver.