# Adding Structured Data and AI-Crawler Rules

> Source: https://agilitycms.com/docs/developers/structured-data-and-ai-crawlers

Search engines and AI assistants understand a page faster when it describes itself. Two things do most of that work: **structured data** (JSON-LD in schema.org vocabulary) and **crawler rules** (`robots.txt`, plus per-page `robots` meta tags).

This article shows how to add both by pasting markup into a page's existing SEO and Scripts fields, and what the Agility starters do with that markup.

> **Using the GEO tab?** Pages and dynamic-page items now have a **GEO** tab with a validated **JSON-LD schema** field, and `@agility/geo` builds one structured-data graph from it, your content and your site settings. If your site is on `@agility/nextjs` 16.1 or later, start with [Rendering SEO & GEO Metadata on Your Site](/docs/developers/rendering-seo-geo-metadata). This article covers sites that render pasted markup, and the `robots.txt` side, which applies either way.

## What you need

- **The JSON-LD has to be in the server-rendered HTML.** Many AI crawlers don't run JavaScript, so structured data added in the browser is invisible to them.
- **Your site has to render the fields.** Agility stores the markup and returns it from the Fetch API. Your front end decides what reaches the page. The [Agility Next.js Starter](/docs/nextjs/how-the-next-js-starter-works) renders it (see [What the starters render](#what-the-starters-render)). For any other site, check your page template.

## Where to paste JSON-LD

Every page has two places that take markup, both on the **SEO** tab of the page's **SEO** panel (select **SEO** in the page's right-hand sidebar):

| Field in Agility | Fetch API field | Use it for |
| --- | --- | --- |
| **Additional Header Markup** | `page.seo.metaHTML` | `<meta>` tags and JSON-LD. **Recommended for JSON-LD.** |
| **Scripts** > **Scripts included in `<head>`** | `page.scripts.top` | JavaScript that must load early, such as a page-specific tag |
| **Scripts** > **Scripts included in `<body>`** | `page.scripts.bottom` | JavaScript that can load after the content, such as a tracking pixel |

Both starters pick up JSON-LD from any of the three fields. Keeping it in **Additional Header Markup** keeps the page's SEO in one place.

Paste each block as a complete tag:

```html
<script type="application/ld+json">
{ ... }
</script>
```

You can paste several blocks into the same field, but one block with an `@graph` (below) is easier to keep consistent.

### Organization and WebSite: on the home page

Describe your organization and your site once, on the home page. Give each one an `@id` (a URL with a fragment, which doesn't have to resolve) so other blocks can refer to it:

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://www.example.com/#organization",
      "name": "Example Co",
      "url": "https://www.example.com/",
      "logo": {
        "@type": "ImageObject",
        "url": "https://www.example.com/images/logo.png"
      },
      "sameAs": [
        "https://www.linkedin.com/company/example-co",
        "https://github.com/example-co"
      ]
    },
    {
      "@type": "WebSite",
      "@id": "https://www.example.com/#website",
      "url": "https://www.example.com/",
      "name": "Example Co",
      "publisher": { "@id": "https://www.example.com/#organization" }
    }
  ]
}
</script>
```

Don't add a `SearchAction` unless your site really has a search results URL that takes a query.

### Article: on a static article page

For a static page that is an article, guide or news story:

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "@id": "https://www.example.com/guides/choosing-a-plan#article",
  "mainEntityOfPage": "https://www.example.com/guides/choosing-a-plan",
  "headline": "How to Choose the Right Plan",
  "description": "What each plan includes and how to decide between them.",
  "image": ["https://www.example.com/images/choosing-a-plan.jpg"],
  "datePublished": "2026-09-01T09:00:00Z",
  "dateModified": "2026-10-10T14:30:00Z",
  "author": {
    "@type": "Person",
    "name": "Jane Doe",
    "url": "https://www.example.com/about/jane-doe"
  },
  "publisher": {
    "@type": "Organization",
    "@id": "https://www.example.com/#organization",
    "name": "Example Co",
    "url": "https://www.example.com/"
  }
}
</script>
```

- `headline` should match the page's visible title.
- Use full ISO 8601 dates with a time zone. Update `dateModified` when the content really changes, not on every publish.
- The `publisher` repeats the organization's `name` and `url` alongside its `@id`, because not every parser follows an `@id` to another page.

### Dynamic pages: generate it in code

A dynamic page (one page per item in a list, such as a blog post) has **one** set of page fields, and the starters render the dynamic page's Additional Header Markup on **every** item it shows. An `Article` pasted there would give every post the same headline and dates.

For dynamic pages, build the JSON-LD in your code from the item's fields instead. [Implementing JSON-LD Structured Data with Next.js](/docs/nextjs/implementing-json-ld-structured-data-with-next-js) walks through `BlogPosting`, `Event` and `Article`, and [Making Your Agility-Powered Site Readable by AI](/docs/developers/making-your-site-readable-by-ai#4-add-json-ld-structured-data) shows an `@graph` with a shared organization `@id`.

Content items in a dynamic page's list also have their own **Additional Header Markup**, returned on the item as `seo.metaHTML`. Both starters render it after the dynamic page's, so a per-item block works there. On your own site, read `dynamicPageItem.seo.metaHTML` in your page and render it the same way. Items also have their own **JSON-LD schema** field on the **GEO** tab, which isn't shared with the dynamic page.

## Keep it valid

A JSON-LD block that doesn't parse is ignored by search engines, so a small typo removes all of it.

- **Write strict JSON.** Double quotes only, no trailing commas, no comments.
- **Watch for pasted characters.** Text copied from a word processor or rich text editor can carry curly quotes (`“ ”`) as JSON delimiters, which break it, and invisible non-breaking spaces or line separators (U+00A0, U+2028), which `JSON.parse` rejects outside strings. Paste through a plain-text editor first. (The Next.js starter replaces those invisible characters before parsing, but other sites may not.)
- **Write characters, not HTML entities.** The content of a `<script>` tag isn't HTML-decoded, so `&amp;` stays `&amp;` in the data. Write `&`.
- **Never write `</script>` inside a value.** It ends the tag early. The starters re-serialize valid JSON-LD and replace every `<` with its JSON Unicode escape, so the tag can't be closed early. If you render JSON-LD yourself, do the same:

  ```ts
  const safe = JSON.stringify(JSON.parse(raw)).replace(/</g, "\\u003c")
  ```

- **Only describe what's on the page.** Structured data that disagrees with the visible content can lose you rich results. FAQ markup in particular now shows as a rich result on Google only for a narrow set of sites, so add `FAQPage` only when the questions and answers are visible on the page.

### Check it

After you publish, check the live page, not the field:

```bash
# Is the JSON-LD in the server HTML?
curl -s https://www.example.com/guides/choosing-a-plan | grep -c 'application/ld+json'
```

Then paste the URL into Google's [Rich Results Test](https://search.google.com/test/rich-results) and the [Schema Markup Validator](https://validator.schema.org/). The Next.js starter logs a `Skipping invalid JSON-LD` warning on the server when a block doesn't parse, so check your build or function logs if a block is missing.

## Page scripts and Global Scripts

**Page scripts** are the two **Scripts** fields above. They belong to one page.

**Global Scripts** (under **Settings** > **Global Scripts**) hold sitewide markup such as an analytics tag. They're output automatically by sites built on the classic .NET `Agility.Web` SDK. **The Fetch API doesn't return Global Scripts**, so on a headless site, sitewide scripts live in your own code, usually the root layout.

Each page has an **Exclude Global Scripts from layout** checkbox in its **Scripts** section, for a page that shouldn't carry the sitewide scripts (a thank-you page you don't want in analytics, for example). The Fetch API returns it as `page.scripts.excludedFromGlobal`. A headless site that wants to honour it can check the flag before rendering its own sitewide scripts:

```tsx
// app/[...slug]/page.tsx (inside the page component)
const excludeGlobal = agilityData.page?.scripts?.excludedFromGlobal === true

return (
  <>
    {!excludeGlobal && <SiteAnalytics />}
    {/* ...the rest of the page */}
  </>
)
```

`page.scripts` is typed on the `Page` type from `@agility/content-fetch` 2.0.13 and later. On an older version, declare it yourself:

```ts
import type { Page } from "@agility/content-fetch"

type PageWithScripts = Page & {
  scripts?: { top: string | null; bottom: string | null; excludedFromGlobal: boolean }
}
```

> **The flag was always `false` in the Fetch API until it was fixed in October 2026.** The Fetch API copy of a page is refreshed when the page is published, so if you rely on the flag, re-publish the pages that have the box ticked. If you saved pages through the Management API before the fix, open them, check the box, and save and publish again.

## What the starters render

| | [Next.js Starter](https://github.com/agility/agilitycms-nextjs-starter) | Vibe-code app starter |
| --- | --- | --- |
| JSON-LD from Additional Header Markup and both Scripts fields | Validated, re-serialized, rendered as `<script type="application/ld+json">` in the server HTML. Invalid blocks are skipped with a warning | Same |
| Other `<script>` tags | Rendered as plain `<script>` elements in document order, keeping their attributes (`async`, `defer`, `src`, `data-*`…). Inline `on*` handlers are dropped | Rendered keeping `type`, `src`, `async` and `defer` |
| Where they go | Additional Header Markup (the dynamic page's, then the item's) and **Scripts included in `<head>`** at the start of the page content, after the site header; **Scripts included in `<body>`** after the page. JSON-LD comes first | Before and after the page template |
| `<meta property="og:…">` and `article:…` | Rendered as `<meta property>` through Next.js metadata | `og:title`, `og:description`, `og:url`, `og:site_name` and `og:image` through Next.js metadata |
| `<meta name="…">` (`robots`, `twitter:*`, `google-site-verification`…) | Rendered as `<meta name>` | Rendered as `<meta name>` |
| The **GEO** tab's fields | `<AgilityJsonLd />` renders one JSON-LD graph (WebPage, BreadcrumbList and the editor's **JSON-LD schema**) before the pasted blocks, and `generateAgilityMetadata()` sets the title, description and canonical URL. See [Rendering SEO & GEO Metadata on Your Site](/docs/developers/rendering-seo-geo-metadata) | Same, plus `hreflang` alternates for each locale |
| A `description` meta tag | The page's **Meta Description** (else its GEO **Summary**) wins; a `description` tag in the markup is used only when both are empty | Same |
| `robots.txt`, `llms.txt` | Not included: add them (below) | Not included |

Two things to know about scripts in the starters:

- **They're in the `<body>`, not the `<head>`.** The App Router doesn't let a page put a blocking script in `<head>`. JSON-LD works anywhere in the document, so this doesn't affect structured data. React moves `<script async src>` tags into `<head>` itself.
- **They run on a full page load only.** Following a link inside the site (client-side navigation) doesn't run the new page's scripts, and coming back doesn't re-run them. A script that has to run on every route change, such as an analytics page view, should listen for navigation itself or be written as a client component.

If you started from an older copy of the starter, its `resolveAgilityMetaData.ts` keeps only `<meta property>` tags and drops everything else, including JSON-LD, with a `Could not output tag in Additional Header Markup` warning. Copy `lib/cms-content/getPageScripts.ts`, `components/common/PageScripts.tsx` and the updated `resolveAgilityMetaData.ts` from the current starter.

## AI-crawler rules

Agility doesn't host `robots.txt` for a headless site: your site serves it. [Controlling Which Crawlers and AI Bots Can Read Your Site](/docs/developers/controlling-crawlers-and-ai-bots) covers which bots do what, the rules for each choice, and a `robots.ts` for the Next.js Starter. Two things to keep in mind:

- Blocking an AI **training** bot (`GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, `Applebot-Extended`) keeps your content out of future training data and doesn't affect search or AI answers. Blocking an AI **search** bot (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) removes you from that assistant's answers. Google's AI Overviews and AI Mode use the normal `Googlebot` crawl.
- The **AI Crawlers** setting on a page's **GEO** tab is stored and returned as `geo.ai.crawl`, but it blocks nothing until your `robots.txt` route reads it.

### Per-page rules

To keep one page out of search results, add a `robots` meta tag to its **Additional Header Markup**. Both starters render it:

```html
<meta name="robots" content="noindex">
```

Google applies its snippet controls to AI Overviews and AI Mode as well, so `<meta name="robots" content="max-snippet:50">` or `nosnippet` limits how much of the page can be quoted there. Use them sparingly: a page that can't be quoted is rarely cited.

Unticking **Visible on Sitemap** only hides the page from your site's sitemap (if your site builds one from the Agility sitemap). It doesn't stop crawlers that find the page through links.

### llms.txt (optional)

[llms.txt](/docs/overview/glossary-llms-txt) is a Markdown index of your important pages for AI agents. It's a proposed convention: search engines don't use it as a ranking signal, but coding agents and some tools read it. It's cheap to add from the Agility sitemap: see [Publish an llms.txt index](/docs/developers/making-your-site-readable-by-ai#2-publish-an-llmstxt-index).

## Native GEO fields

Editors can now set a page type, a summary and validated JSON-LD on the **GEO** tab instead of pasting markup. [Rendering SEO & GEO Metadata on Your Site](/docs/developers/rendering-seo-geo-metadata) shows how to render them. Markup in the fields above keeps working.

## Related

- [Controlling Which Crawlers and AI Bots Can Read Your Site](/docs/developers/controlling-crawlers-and-ai-bots)
- [Rendering SEO & GEO Metadata on Your Site](/docs/developers/rendering-seo-geo-metadata)
- [Making Your Agility-Powered Site Readable by AI](/docs/developers/making-your-site-readable-by-ai)
- [Implementing JSON-LD Structured Data with Next.js](/docs/nextjs/implementing-json-ld-structured-data-with-next-js)
- [Managing SEO for Developers](/docs/developers/manging-seo-for-developers)
- [Managing SEO for Editors](/docs/editors/manage-seo)
- [Faster indexing with XML sitemaps and IndexNow](/docs/developers/faster-indexing-with-xml-sitemaps-and-indexnow)
- [Website Deployment Checklist](/docs/developers/website-deployment-checklist)

---

## Related pages

- [Website Deployment Checklist](https://agilitycms.com/docs/developers/website-deployment-checklist.md)
- [Legacy Content Migration](https://agilitycms.com/docs/developers/legacy-content-migration.md)
- [Faster indexing with XML sitemaps and IndexNow](https://agilitycms.com/docs/developers/faster-indexing-with-xml-sitemaps-and-indexnow.md)
- [Making Your Agility-Powered Site Readable by AI](https://agilitycms.com/docs/developers/making-your-site-readable-by-ai.md)
- [Controlling Which Crawlers and AI Bots Can Read Your Site](https://agilitycms.com/docs/developers/controlling-crawlers-and-ai-bots.md)
- [Improving and Monitoring Lighthouse Scores](https://agilitycms.com/docs/developers/improving-and-monitoring-lighthouse-scores.md)
- [Rendering SEO & GEO Metadata on Your Site](https://agilitycms.com/docs/developers/rendering-seo-geo-metadata.md)

Index of every docs page: https://agilitycms.com/docs/llms.txt