What is your developer-dependent CMS actually costing you? Calculate your costs and get a full report
What is your developer-dependent CMS actually costing you? Calculate your costs and get a full report
Going Live
Choose which search engines and AI bots can read your Agility-powered site: training, search and user-fetch bots, robots.txt rules for each choice, a robots.ts for the Next.js Starter, per-page options, and what robots.txt can't enforce.
Search engines and AI companies send bots to read your site. Some build a search index, some collect training data for AI models, and some fetch a single page because a person asked an assistant a question. You decide which of them are welcome with a robots.txt file at the root of your domain.
Agility is headless, so it doesn't publish a robots.txt for you: your site serves it. This guide covers which bots do what, the rules for the common choices, how to serve the file from the Agility Next.js Starter, and what robots.txt can't do.
The short version. Blocking AI training bots keeps your content out of future training data and doesn't affect search. Blocking AI search bots removes you from AI answers. Most sites that want to opt out of training should block only the training bots.
AI companies run separate bots for separate jobs, and each one obeys its own User-agent group in robots.txt. Bot names as of October 2026:
| Job | User agents | Blocking it… |
|---|---|---|
| Training: collects content that may be used to train AI models | GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, a public dataset many models train on), Google-Extended, Applebot-Extended | opts your content out of future training. It doesn't affect search results. |
| Search index: indexes pages so an assistant can find and cite them | OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot | removes your pages from that assistant's answers. |
| User fetch: opens one page because a person asked about it | ChatGPT-User, Claude-User, Perplexity-User | stops Claude-User. OpenAI and Perplexity say these fetches may ignore robots.txt, because a person asked for the page. |
Two of these are tokens rather than bots. Google-Extended and Applebot-Extended never visit your site; Google and Apple read their own crawlers' visits and check these tokens to decide whether the content may be used for training. Blocking them doesn't remove you from Google Search, from Google's AI Overviews and AI Mode, or from Siri and Spotlight. Those follow Googlebot and Applebot, and blocking either of those removes you from that search engine entirely.
The vendors add and rename bots. Check their current lists before you rely on a name: OpenAI, Anthropic, Perplexity, Google and Apple.
A robots.txt is a list of groups. Each group names one or more user agents and the paths they may not read. A bot follows the most specific group that names it and ignores the rest, so a bot named in its own group doesn't also obey User-agent: *.
This is what a site without a robots.txt already gets. Write it out anyway, so the file can point to your sitemap:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
The choice most sites mean when they ask to "block AI":
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Search engines and AI search bots fall through to the * group and keep reading the site.
Add the search and user-fetch bots to the blocked group:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /
User-agent: *
Allow: /
Your pages will stop appearing in ChatGPT search, Claude and Perplexity answers. Google's AI Overviews still use your pages, because they come from Googlebot.
Replace Disallow: / with the paths to protect. A rule matches every URL that starts with it:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /research/
Disallow: /reports/
User-agent: *
Allow: /
Don't use robots.txt to hide private content. Anyone can read the file, so it lists the paths you'd rather nobody found, and a bot that ignores it can still read them. Put private content behind authentication.
The Agility Next.js Starter doesn't include a robots.txt. A request for /robots.txt reaches the catch-all page route and returns a 404, which crawlers treat as "everything allowed".
Add a robots.ts file to the app folder. Next.js serves its output at /robots.txt:
// app/robots.ts
import type { MetadataRoute } from "next"
const SITE_URL = "https://www.example.com"
// Bots that collect AI training data. Blocking them doesn't affect search.
const AI_TRAINING_BOTS = ["GPTBot", "ClaudeBot", "CCBot", "Google-Extended", "Applebot-Extended"]
export default function robots(): MetadataRoute.Robots {
// Keep preview and staging deployments out of every index
if (process.env.VERCEL_ENV && process.env.VERCEL_ENV !== "production") {
return { rules: { userAgent: "*", disallow: "/" } }
}
return {
rules: [
{ userAgent: AI_TRAINING_BOTS, disallow: "/" },
{ userAgent: "*", allow: "/" },
],
// Only list a sitemap your site actually serves
sitemap: `${SITE_URL}/sitemap.xml`,
}
}
A few things to know:
app/robots.ts before app/[...slug], so you don't need to change the page route. The starter's proxy.ts matches /robots.txt but only acts on preview and ContentID requests, so it passes the request through. If you've customized your proxy to rewrite every path, exclude /robots.txt from it.VERCEL_ENV check blocks all crawlers on preview deployments, which covers the temporary domain in the Website Deployment Checklist. On another host, use that host's equivalent variable or your own.robots.ts once at build time, so a change to the rules needs a new deployment. That's usually what you want for a file that changes rarely.public/robots.txt is fine if you don't need logic. Use one or the other, not both.On other frameworks, serve the same text at /robots.txt with a text/plain content type. In ASP.NET Core, a robots.txt file in wwwroot is served by the static files middleware.
Crawlers only read robots.txt at the root of a host: https://www.example.com/robots.txt. A file at /blog/robots.txt is ignored. If your Agility site is served under a path of another domain (the way these docs are served at agilitycms.com/docs), the main site's robots.txt controls crawling, and that's where the rules have to go. Each subdomain needs its own file.
Path rules in robots.txt work for sections. For one page, the choice depends on what you want:
Disallow line for its path to the training-bot group.<meta name="robots" content="noindex"> to the page's Additional Header Markup, in its SEO properties. A search engine has to be allowed to read the page to see this tag, so don't also disallow the page in robots.txt.The current Next.js Starter outputs <meta name> tags from Additional Header Markup. Copies of the starter from before October 2026 only output <meta property> tags and drop name tags with a console warning. If you started from an older copy, compare your lib/cms-content/resolveAgilityMetaData.ts with the current starter.
Unchecking Visible on Sitemap doesn't block crawlers by itself. It's a flag your site's code reads: it usually takes the page out of sitemap.xml, but a crawler that finds a link to the page still reads it unless your code also adds a noindex tag for hidden pages. Check what yours does.
robots.txt is a request, not a lock. The major search and AI crawlers honor it, and some don't:
robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.To enforce a block, refuse the request before it reaches your site, with your host's firewall or CDN: a custom firewall rule that matches the User-Agent header (Vercel and most CDNs support these), or a managed AI-bot blocking setting where your CDN offers one, such as Cloudflare's. A user agent is easy to fake, so the managed settings that verify bots by IP address are more reliable than matching names yourself. Keep /robots.txt itself reachable: Anthropic notes that blocking its bots by IP address stops them from reading the opt-out in your robots.txt.
If you also serve an llms.txt index or Markdown copies of your pages (see Making Your Agility-Powered Site Readable by AI), the same rules apply to those URLs. A bot you disallow from / won't read /llms.txt either. If you block AI bots from most of your site but want agents to find some of it, leave those paths and /llms.txt allowed.
# 1. The file is served at the root, as plain text, with a 200
curl -sI https://www.example.com/robots.txt
# 2. The rules are the ones you deployed
curl -s https://www.example.com/robots.txt
# 3. A preview deployment blocks everything
curl -s https://my-site-git-feature.vercel.app/robots.txt
Then open the robots.txt report in Google Search Console to see the version Google last fetched and any lines it couldn't parse. Your server logs or analytics show which bots visit and which paths they request. Check them a week or two after a change: a blocked bot that keeps requesting pages is one to stop at the firewall.