AI Crawlers and robots.txt: Deciding What Bots Can Read

For years, most website owners only thought about one kind of visitor bot: search engine crawlers. Now AI companies run their own crawlers too, and they read the web for different reasons. Some collect content to train AI models. Some fetch pages so an AI search tool can answer questions with links. Some visit a page only when a user asks an assistant to read it.

You can express preferences about many of these bots in a small file called robots.txt. Here is how it works and how to decide what fits your organization.

What robots.txt does

robots.txt is a plain text file at the root of your website, at your domain followed by /robots.txt. It tells crawlers which parts of the site they may or may not crawl, using groups of rules that name a crawler (the user agent) and list paths to allow or disallow.

For example, a group that blocks a crawler named ExampleBot from the whole site looks like this, with each rule on its own line in the file:

User-agent: ExampleBot

Disallow: /

A few important facts:

• It is voluntary. Reputable companies say their crawlers follow robots.txt, but malicious bots can simply ignore it.
• It is not security. It does not hide or protect content. Anything private should require a login.
• Blocking crawling is not the same as blocking indexing. Google notes that a page blocked by robots.txt can still appear in results if other pages link to it. To keep a page out of Google's index, use a noindex instruction instead and allow crawling so Google can see it.

Google's introduction to robots.txt explains the rules in detail.

Three kinds of AI crawlers

AI companies typically separate their bots by purpose:

1. Training crawlers collect public web content that may be used to train AI models.
2. Search crawlers index pages so an AI search product can show and link to them in answers.
3. User-initiated fetchers load a page when a person asks an assistant to read or summarize a specific link.

This separation matters because you might feel differently about each. Many organizations are happy to appear, with a link, in AI search answers but would rather not contribute content to model training.

Examples of crawler names

Crawler names and purposes change, so always check each company's current documentation before writing rules. As examples:

• OpenAI documents GPTBot for training, OAI-SearchBot for search features and ChatGPT-User for actions users request.
• Google uses Googlebot for Google Search, including its AI features in Search. It also offers a separate token, Google-Extended, that lets sites control whether content is used for its Gemini models; Google says this does not affect inclusion in Search.
• Anthropic documents ClaudeBot along with separate agents for search and user requests.
• Perplexity documents PerplexityBot for its search index and a separate agent for user requests.
• Apple offers Applebot-Extended to control use of content for training its AI models.
• Common Crawl's CCBot builds an open web archive that many AI projects have used.

A decision framework

Start with your goal, then choose rules that match it.

Goal: be found and cited as widely as possible

Allow search engines and AI search crawlers. This is often the right choice for small businesses, churches and nonprofits that want people to find them. Our guide to generative engine optimization covers how to make your content easy to cite.

Goal: be found in AI search, but limit model training

Allow search crawlers and user-initiated fetchers, and disallow training crawlers and training control tokens such as Google-Extended and Applebot-Extended. Keep Googlebot allowed if you want to remain in Google Search.

Goal: keep content away from AI systems

Disallow the AI crawlers you can identify, understanding that this may also remove you from AI search answers and that robots.txt cannot stop bots that ignore it. Content that must stay private belongs behind a login.

Practical tips

• Never block Googlebot or Bingbot by accident if you want to appear in search.
• Test changes with Search Console's robots.txt report after editing. Our guide to setting up Google Search Console shows where to find it.
• Check your platform's settings. Some website platforms, hosts and content delivery networks offer built-in options to manage AI crawlers, which may be easier than editing the file.
• Watch for heavy crawling. If a bot overloads your site, your host or a firewall service can help with rate limits.
• Revisit your choices periodically, as companies add or rename crawlers.

A note about llms.txt

You may hear about llms.txt, a proposed file that summarizes a site for AI tools. It is an informal proposal rather than an established standard, and Google has said it does not use it for Search. Adding one is optional and should not replace good content and sensible crawler rules.

Frequently asked questions

Will blocking AI crawlers hurt my Google rankings?

Blocking AI-specific crawlers or the Google-Extended token does not, according to Google, affect your inclusion in Google Search. Blocking Googlebot itself would remove your pages from Google Search, so be careful not to do that unintentionally.

Can robots.txt stop all AI use of my content?

No. It only guides crawlers that choose to follow it, and it cannot remove content already collected. For truly private content, require a login.

Where is my robots.txt file?

Visit your domain followed by /robots.txt in a browser. Many website platforms generate it automatically and provide settings to change it; otherwise it is a file at the top level of your site.