Adding Structured Data and AI-Crawler Rules

Search engines and AI assistants understand a page faster when it describes itself. Two things do most of that work: structured data (JSON-LD in schema.org vocabulary) and crawler rules (robots.txt, plus per-page robots meta tags).

This article shows how to add both by pasting markup into a page's existing SEO and Scripts fields, and what the Agility starters do with that markup.

Using the GEO tab? Pages and dynamic-page items now have a GEO tab with a validated JSON-LD schema field, and @agility/geo builds one structured-data graph from it, your content and your site settings. If your site is on @agility/nextjs 16.1 or later, start with Rendering SEO & GEO Metadata on Your Site. This article covers sites that render pasted markup, and the robots.txt side, which applies either way.

What you need

• The JSON-LD has to be in the server-rendered HTML. Many AI crawlers don't run JavaScript, so structured data added in the browser is invisible to them.
• Your site has to render the fields. Agility stores the markup and returns it from the Fetch API. Your front end decides what reaches the page. The Agility Next.js Starter renders it (see What the starters render). For any other site, check your page template.

Where to paste JSON-LD

Every page has two places that take markup, both on the SEO tab of the page's SEO panel (select SEO in the page's right-hand sidebar):

Field in Agility	Fetch API field	Use it for
Additional Header Markup	page.seo.metaHTML	<meta> tags and JSON-LD. Recommended for JSON-LD.
Scripts > Scripts included in <head>	page.scripts.top	JavaScript that must load early, such as a page-specific tag
Scripts > Scripts included in <body>	page.scripts.bottom	JavaScript that can load after the content, such as a tracking pixel

Both starters pick up JSON-LD from any of the three fields. Keeping it in Additional Header Markup keeps the page's SEO in one place.

Paste each block as a complete tag:

<script type="application/ld+json">
{ ... }
</script>

You can paste several blocks into the same field, but one block with an @graph (below) is easier to keep consistent.

Organization and WebSite: on the home page

Describe your organization and your site once, on the home page. Give each one an @id (a URL with a fragment, which doesn't have to resolve) so other blocks can refer to it:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://www.example.com/#organization",
      "name": "Example Co",
      "url": "https://www.example.com/",
      "logo": {
        "@type": "ImageObject",
        "url": "https://www.example.com/images/logo.png"
      },
      "sameAs": [
        "https://www.linkedin.com/company/example-co",
        "https://github.com/example-co"
      ]
    },
    {
      "@type": "WebSite",
      "@id": "https://www.example.com/#website",
      "url": "https://www.example.com/",
      "name": "Example Co",
      "publisher": { "@id": "https://www.example.com/#organization" }
    }
  ]
}
</script>

Don't add a SearchAction unless your site really has a search results URL that takes a query.

Article: on a static article page

For a static page that is an article, guide or news story:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "@id": "https://www.example.com/guides/choosing-a-plan#article",
  "mainEntityOfPage": "https://www.example.com/guides/choosing-a-plan",
  "headline": "How to Choose the Right Plan",
  "description": "What each plan includes and how to decide between them.",
  "image": ["https://www.example.com/images/choosing-a-plan.jpg"],
  "datePublished": "2026-09-01T09:00:00Z",
  "dateModified": "2026-10-10T14:30:00Z",
  "author": {
    "@type": "Person",
    "name": "Jane Doe",
    "url": "https://www.example.com/about/jane-doe"
  },
  "publisher": {
    "@type": "Organization",
    "@id": "https://www.example.com/#organization",
    "name": "Example Co",
    "url": "https://www.example.com/"
  }
}
</script>

• headline should match the page's visible title.
• Use full ISO 8601 dates with a time zone. Update dateModified when the content really changes, not on every publish.
• The publisher repeats the organization's name and url alongside its @id, because not every parser follows an @id to another page.

Dynamic pages: generate it in code

A dynamic page (one page per item in a list, such as a blog post) has one set of page fields, and the starters render the dynamic page's Additional Header Markup on every item it shows. An Article pasted there would give every post the same headline and dates.

For dynamic pages, build the JSON-LD in your code from the item's fields instead. Implementing JSON-LD Structured Data with Next.js walks through BlogPosting, Event and Article, and Making Your Agility-Powered Site Readable by AI shows an @graph with a shared organization @id.

Content items in a dynamic page's list also have their own Additional Header Markup, returned on the item as seo.metaHTML. Both starters render it after the dynamic page's, so a per-item block works there. On your own site, read dynamicPageItem.seo.metaHTML in your page and render it the same way. Items also have their own JSON-LD schema field on the GEO tab, which isn't shared with the dynamic page.

Keep it valid

A JSON-LD block that doesn't parse is ignored by search engines, so a small typo removes all of it.

• Write strict JSON. Double quotes only, no trailing commas, no comments.

• Watch for pasted characters. Text copied from a word processor or rich text editor can carry curly quotes (“ ”) as JSON delimiters, which break it, and invisible non-breaking spaces or line separators (U+00A0, U+2028), which JSON.parse rejects outside strings. Paste through a plain-text editor first. (The Next.js starter replaces those invisible characters before parsing, but other sites may not.)

• Write characters, not HTML entities. The content of a <script> tag isn't HTML-decoded, so & stays & in the data. Write &.

• Never write </script> inside a value. It ends the tag early. The starters re-serialize valid JSON-LD and replace every < with its JSON Unicode escape, so the tag can't be closed early. If you render JSON-LD yourself, do the same:

  const safe = JSON.stringify(JSON.parse(raw)).replace(/</g, "\\u003c")
  

• Only describe what's on the page. Structured data that disagrees with the visible content can lose you rich results. FAQ markup in particular now shows as a rich result on Google only for a narrow set of sites, so add FAQPage only when the questions and answers are visible on the page.

Check it

After you publish, check the live page, not the field:

# Is the JSON-LD in the server HTML?
curl -s https://www.example.com/guides/choosing-a-plan | grep -c 'application/ld+json'

Then paste the URL into Google's Rich Results Test and the Schema Markup Validator. The Next.js starter logs a Skipping invalid JSON-LD warning on the server when a block doesn't parse, so check your build or function logs if a block is missing.

Page scripts and Global Scripts

Page scripts are the two Scripts fields above. They belong to one page.

Global Scripts (under Settings > Global Scripts) hold sitewide markup such as an analytics tag. They're output automatically by sites built on the classic .NET Agility.Web SDK. The Fetch API doesn't return Global Scripts, so on a headless site, sitewide scripts live in your own code, usually the root layout.

Each page has an Exclude Global Scripts from layout checkbox in its Scripts section, for a page that shouldn't carry the sitewide scripts (a thank-you page you don't want in analytics, for example). The Fetch API returns it as page.scripts.excludedFromGlobal. A headless site that wants to honour it can check the flag before rendering its own sitewide scripts:

// app/[...slug]/page.tsx (inside the page component)
const excludeGlobal = agilityData.page?.scripts?.excludedFromGlobal === true

return (
  <>
    {!excludeGlobal && <SiteAnalytics />}
    {/* ...the rest of the page */}
  </>
)

page.scripts is typed on the Page type from @agility/content-fetch 2.0.13 and later. On an older version, declare it yourself:

import type { Page } from "@agility/content-fetch"

type PageWithScripts = Page & {
  scripts?: { top: string | null; bottom: string | null; excludedFromGlobal: boolean }
}

The flag was always false in the Fetch API until it was fixed in October 2026. The Fetch API copy of a page is refreshed when the page is published, so if you rely on the flag, re-publish the pages that have the box ticked. If you saved pages through the Management API before the fix, open them, check the box, and save and publish again.

What the starters render

	Next.js Starter	Vibe-code app starter
JSON-LD from Additional Header Markup and both Scripts fields	Validated, re-serialized, rendered as <script type="application/ld+json"> in the server HTML. Invalid blocks are skipped with a warning	Same
Other <script> tags	Rendered as plain <script> elements in document order, keeping their attributes (async, defer, src, data-*…). Inline on* handlers are dropped	Rendered keeping type, src, async and defer
Where they go	Additional Header Markup (the dynamic page's, then the item's) and Scripts included in <head> at the start of the page content, after the site header; Scripts included in <body> after the page. JSON-LD comes first	Before and after the page template
<meta property="og:…"> and article:…	Rendered as <meta property> through Next.js metadata	og:title, og:description, og:url, og:site_name and og:image through Next.js metadata
<meta name="…"> (robots, twitter:*, google-site-verification…)	Rendered as <meta name>	Rendered as <meta name>
The GEO tab's fields	<AgilityJsonLd /> renders one JSON-LD graph (WebPage, BreadcrumbList and the editor's JSON-LD schema) before the pasted blocks, and generateAgilityMetadata() sets the title, description and canonical URL. See Rendering SEO & GEO Metadata on Your Site	Same, plus hreflang alternates for each locale
A description meta tag	The page's Meta Description (else its GEO Summary) wins; a description tag in the markup is used only when both are empty	Same
robots.txt, llms.txt	Not included: add them (below)	Not included

Two things to know about scripts in the starters:

• They're in the <body>, not the <head>. The App Router doesn't let a page put a blocking script in <head>. JSON-LD works anywhere in the document, so this doesn't affect structured data. React moves <script async src> tags into <head> itself.
• They run on a full page load only. Following a link inside the site (client-side navigation) doesn't run the new page's scripts, and coming back doesn't re-run them. A script that has to run on every route change, such as an analytics page view, should listen for navigation itself or be written as a client component.

If you started from an older copy of the starter, its resolveAgilityMetaData.ts keeps only <meta property> tags and drops everything else, including JSON-LD, with a Could not output tag in Additional Header Markup warning. Copy lib/cms-content/getPageScripts.ts, components/common/PageScripts.tsx and the updated resolveAgilityMetaData.ts from the current starter.

AI-crawler rules

Agility doesn't host robots.txt for a headless site: your site serves it. Controlling Which Crawlers and AI Bots Can Read Your Site covers which bots do what, the rules for each choice, and a robots.ts for the Next.js Starter. Two things to keep in mind:

• Blocking an AI training bot (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) keeps your content out of future training data and doesn't affect search or AI answers. Blocking an AI search bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from that assistant's answers. Google's AI Overviews and AI Mode use the normal Googlebot crawl.
• The AI Crawlers setting on a page's GEO tab is stored and returned as geo.ai.crawl, but it blocks nothing until your robots.txt route reads it.

Per-page rules

To keep one page out of search results, add a robots meta tag to its Additional Header Markup. Both starters render it:

<meta name="robots" content="noindex">

Google applies its snippet controls to AI Overviews and AI Mode as well, so <meta name="robots" content="max-snippet:50"> or nosnippet limits how much of the page can be quoted there. Use them sparingly: a page that can't be quoted is rarely cited.

Unticking Visible on Sitemap only hides the page from your site's sitemap (if your site builds one from the Agility sitemap). It doesn't stop crawlers that find the page through links.

llms.txt (optional)

llms.txt is a Markdown index of your important pages for AI agents. It's a proposed convention: search engines don't use it as a ranking signal, but coding agents and some tools read it. It's cheap to add from the Agility sitemap: see Publish an llms.txt index.

Native GEO fields

Editors can now set a page type, a summary and validated JSON-LD on the GEO tab instead of pasting markup. Rendering SEO & GEO Metadata on Your Site shows how to render them. Markup in the fields above keeps working.

Related

• Controlling Which Crawlers and AI Bots Can Read Your Site
• Rendering SEO & GEO Metadata on Your Site
• Making Your Agility-Powered Site Readable by AI
• Implementing JSON-LD Structured Data with Next.js
• Managing SEO for Developers
• Managing SEO for Editors
• Faster indexing with XML sitemaps and IndexNow
• Website Deployment Checklist