# Controlling Which Crawlers and AI Bots Can Read Your Site

> Source: https://agilitycms.com/docs/developers/controlling-crawlers-and-ai-bots

Search engines and AI companies send bots to read your site. Some build a search index, some collect training data for AI models, and some fetch a single page because a person asked an assistant a question. You decide which of them are welcome with a `robots.txt` file at the root of your domain.

Agility is headless, so it doesn't publish a `robots.txt` for you: your site serves it. This guide covers which bots do what, the rules for the common choices, how to serve the file from the Agility Next.js Starter, and what `robots.txt` can't do.

> **The short version.** Blocking AI *training* bots keeps your content out of future training data and doesn't affect search. Blocking AI *search* bots removes you from AI answers. Most sites that want to opt out of training should block only the training bots.

## Which Bots Do What

AI companies run separate bots for separate jobs, and each one obeys its own `User-agent` group in `robots.txt`. Bot names as of October 2026:

| Job | User agents | Blocking it… |
| --- | --- | --- |
| **Training**: collects content that may be used to train AI models | `GPTBot` (OpenAI), `ClaudeBot` (Anthropic), `CCBot` (Common Crawl, a public dataset many models train on), `Google-Extended`, `Applebot-Extended` | opts your content out of future training. It doesn't affect search results. |
| **Search index**: indexes pages so an assistant can find and cite them | `OAI-SearchBot` (ChatGPT search), `Claude-SearchBot`, `PerplexityBot` | removes your pages from that assistant's answers. |
| **User fetch**: opens one page because a person asked about it | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | stops `Claude-User`. OpenAI and Perplexity say these fetches may ignore `robots.txt`, because a person asked for the page. |

Two of these are tokens rather than bots. `Google-Extended` and `Applebot-Extended` never visit your site; Google and Apple read their own crawlers' visits and check these tokens to decide whether the content may be used for training. Blocking them doesn't remove you from Google Search, from Google's AI Overviews and AI Mode, or from Siri and Spotlight. Those follow `Googlebot` and `Applebot`, and blocking either of those removes you from that search engine entirely.

The vendors add and rename bots. Check their current lists before you rely on a name: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518), [Perplexity](https://docs.perplexity.ai/guides/bots), [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) and [Apple](https://support.apple.com/en-us/119829).

## Rules for the Common Choices

A `robots.txt` is a list of groups. Each group names one or more user agents and the paths they may not read. A bot follows the most specific group that names it and ignores the rest, so a bot named in its own group doesn't also obey `User-agent: *`.

### Allow everything

This is what a site without a `robots.txt` already gets. Write it out anyway, so the file can point to your sitemap:

```txt
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml
```

### Opt out of AI training, stay in search and AI answers

The choice most sites mean when they ask to "block AI":

```txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml
```

Search engines and AI search bots fall through to the `*` group and keep reading the site.

### Block all AI bots

Add the search and user-fetch bots to the blocked group:

```txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /

User-agent: *
Allow: /
```

Your pages will stop appearing in ChatGPT search, Claude and Perplexity answers. Google's AI Overviews still use your pages, because they come from `Googlebot`.

### Keep AI bots out of one section

Replace `Disallow: /` with the paths to protect. A rule matches every URL that starts with it:

```txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /research/
Disallow: /reports/

User-agent: *
Allow: /
```

Don't use `robots.txt` to hide private content. Anyone can read the file, so it lists the paths you'd rather nobody found, and a bot that ignores it can still read them. Put private content behind authentication.

## Serve robots.txt from the Next.js Starter

The [Agility Next.js Starter](/docs/nextjs/how-the-next-js-starter-works) doesn't include a `robots.txt`. A request for `/robots.txt` reaches the catch-all page route and returns a 404, which crawlers treat as "everything allowed".

Add a [`robots.ts`](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) file to the `app` folder. Next.js serves its output at `/robots.txt`:

```ts
// app/robots.ts
import type { MetadataRoute } from "next"

const SITE_URL = "https://www.example.com"

// Bots that collect AI training data. Blocking them doesn't affect search.
const AI_TRAINING_BOTS = ["GPTBot", "ClaudeBot", "CCBot", "Google-Extended", "Applebot-Extended"]

export default function robots(): MetadataRoute.Robots {
  // Keep preview and staging deployments out of every index
  if (process.env.VERCEL_ENV && process.env.VERCEL_ENV !== "production") {
    return { rules: { userAgent: "*", disallow: "/" } }
  }

  return {
    rules: [
      { userAgent: AI_TRAINING_BOTS, disallow: "/" },
      { userAgent: "*", allow: "/" },
    ],
    // Only list a sitemap your site actually serves
    sitemap: `${SITE_URL}/sitemap.xml`,
  }
}
```

A few things to know:

- **The special file wins over the catch-all.** Next.js matches `app/robots.ts` before `app/[...slug]`, so you don't need to change the page route. The starter's `proxy.ts` matches `/robots.txt` but only acts on preview and `ContentID` requests, so it passes the request through. If you've customized your proxy to rewrite every path, exclude `/robots.txt` from it.
- **The `VERCEL_ENV` check** blocks all crawlers on preview deployments, which covers the temporary domain in the [Website Deployment Checklist](/docs/developers/website-deployment-checklist). On another host, use that host's equivalent variable or your own.
- **It's static.** Next.js builds `robots.ts` once at build time, so a change to the rules needs a new deployment. That's usually what you want for a file that changes rarely.
- **A static file works too.** A plain `public/robots.txt` is fine if you don't need logic. Use one or the other, not both.

On other frameworks, serve the same text at `/robots.txt` with a `text/plain` content type. In ASP.NET Core, a `robots.txt` file in `wwwroot` is served by the static files middleware.

### Only the root counts

Crawlers only read `robots.txt` at the root of a host: `https://www.example.com/robots.txt`. A file at `/blog/robots.txt` is ignored. If your Agility site is served under a path of another domain (the way these docs are served at `agilitycms.com/docs`), the main site's `robots.txt` controls crawling, and that's where the rules have to go. Each subdomain needs its own file.

## Rules for a Single Page

Path rules in `robots.txt` work for sections. For one page, the choice depends on what you want:

- **Keep it out of AI training data:** add a `Disallow` line for its path to the training-bot group.
- **Keep it out of search results:** add `<meta name="robots" content="noindex">` to the page's **Additional Header Markup**, in its SEO properties. A search engine has to be allowed to read the page to see this tag, so don't also disallow the page in `robots.txt`.

The current Next.js Starter outputs `<meta name>` tags from Additional Header Markup. Copies of the starter from before October 2026 only output `<meta property>` tags and drop `name` tags with a console warning. If you started from an older copy, compare your `lib/cms-content/resolveAgilityMetaData.ts` with the [current starter](https://github.com/agility/agilitycms-nextjs-starter).

Unchecking **Visible on Sitemap** doesn't block crawlers by itself. It's a flag your site's code reads: it usually takes the page out of `sitemap.xml`, but a crawler that finds a link to the page still reads it unless your code also adds a `noindex` tag for hidden pages. Check what yours does.

## What robots.txt Can't Do

`robots.txt` is a request, not a lock. The major search and AI crawlers honor it, and some don't:

- **User-triggered fetches may ignore it.** OpenAI says `robots.txt` rules may not apply to `ChatGPT-User`, and Perplexity says `Perplexity-User` generally ignores them.
- **Unknown and badly behaved bots ignore it.** So do bots that claim another bot's user agent.
- **Content already collected stays collected.** Blocking a training bot affects what it collects from now on.

To enforce a block, refuse the request before it reaches your site, with your host's firewall or CDN: a custom firewall rule that matches the `User-Agent` header (Vercel and most CDNs support these), or a managed AI-bot blocking setting where your CDN offers one, such as Cloudflare's. A user agent is easy to fake, so the managed settings that verify bots by IP address are more reliable than matching names yourself. Keep `/robots.txt` itself reachable: Anthropic notes that blocking its bots by IP address stops them from reading the opt-out in your `robots.txt`.

## llms.txt and Blocked Bots

If you also serve an `llms.txt` index or Markdown copies of your pages (see [Making Your Agility-Powered Site Readable by AI](/docs/developers/making-your-site-readable-by-ai)), the same rules apply to those URLs. A bot you disallow from `/` won't read `/llms.txt` either. If you block AI bots from most of your site but want agents to find some of it, leave those paths and `/llms.txt` allowed.

## Check It

```bash
# 1. The file is served at the root, as plain text, with a 200
curl -sI https://www.example.com/robots.txt

# 2. The rules are the ones you deployed
curl -s https://www.example.com/robots.txt

# 3. A preview deployment blocks everything
curl -s https://my-site-git-feature.vercel.app/robots.txt
```

Then open the **robots.txt report** in Google Search Console to see the version Google last fetched and any lines it couldn't parse. Your server logs or analytics show which bots visit and which paths they request. Check them a week or two after a change: a blocked bot that keeps requesting pages is one to stop at the firewall.

## Related

- [Making Your Agility-Powered Site Readable by AI](/docs/developers/making-your-site-readable-by-ai)
- [Website Deployment Checklist](/docs/developers/website-deployment-checklist)
- [Faster Indexing with XML Sitemaps and IndexNow](/docs/developers/faster-indexing-with-xml-sitemaps-and-indexnow)
- [Managing SEO for Developers](/docs/developers/manging-seo-for-developers)
- [How the Next.js Starter Works](/docs/nextjs/how-the-next-js-starter-works)

---

## Related pages

- [Website Deployment Checklist](https://agilitycms.com/docs/developers/website-deployment-checklist.md)
- [Legacy Content Migration](https://agilitycms.com/docs/developers/legacy-content-migration.md)
- [Faster indexing with XML sitemaps and IndexNow](https://agilitycms.com/docs/developers/faster-indexing-with-xml-sitemaps-and-indexnow.md)
- [Making Your Agility-Powered Site Readable by AI](https://agilitycms.com/docs/developers/making-your-site-readable-by-ai.md)

Index of every docs page: https://agilitycms.com/docs/llms.txt