# Faster Indexing with XML Sitemaps and IndexNow

> Source: https://agilitycms.com/docs/developers/faster-indexing-with-xml-sitemaps-and-indexnow

Two complementary techniques get your content discovered faster:

1. **An honest `sitemap.xml`** — every URL reports the date its content *actually* changed, so Google spends its crawl budget on what's new instead of re-checking what isn't.
2. **IndexNow** — a publish-time ping that tells Bing, Yandex, Seznam, Naver (and the AI search that rides on Bing's index) that a URL changed, without waiting for a crawl.

Think of them as **pull** and **push**. The sitemap makes Google's own crawling efficient and trustworthy across your whole site; IndexNow pushes individual changes out instantly to the engines that support it. You want both.

## Why this matters

Google only trusts your sitemap dates if they're consistent with what it finds when it crawls. A sitemap that stamps every URL with the current time — the classic `lastModified: new Date()` bug — trains Google to **ignore** your `lastmod` entirely. An accurate one does the opposite: it earns the signal, and Google can skip the thousand unchanged URLs to spend its budget on the handful that actually changed.

Two things worth internalizing up front:

- **Google uses `lastmod` (when it's trustworthy) and nothing else from your sitemap.** It ignores `<changefreq>` and `<priority>` — Google's own John Mueller and Gary Illyes have said so repeatedly. Don't emit them; a blanket `changefreq` of `"daily"` on every URL just contradicts the accurate `lastmod` you worked to produce.
- **No date is better than a wrong date.** If you can't determine a reliable modified date for a URL, omit `<lastmod>` for that entry rather than inventing one.

## Before you begin

You'll need:

- An Agility CMS site on Next.js (App Router) using `@agility/content-fetch`.
- The standard cached content wrappers most Agility starters ship with — `getSitemapFlat`, `getContentItem`, `getContentList` — each setting a Next.js cache tag and `revalidate`.
- A publish webhook route (e.g. `app/api/revalidate/route.ts`) that Agility calls on publish. You'll extend it for IndexNow.

---

## Part 1 — An accurate sitemap `lastmod`

### Where the dates come from

The flat sitemap Agility returns is a dictionary of `path → node`. Each node has `pageID`, `path`, `visible`, `redirect`, `isFolder`, and — for dynamic pages — a `contentID`. What it does **not** contain is a modified date. So `lastmod` has to be assembled from the content behind each URL:

- **Dynamic pages** (the node has a `contentID`): use the **backing content item's** `properties.modified` — the date the content itself changed.
- **Static pages** (no `contentID`): use the **most recent** of the page object's `properties.modified` and every module/content item on the page. Publishing new content into a page should bump its date, even if the page object was untouched.

### Step 1: A cached `getPage` wrapper

You need the page *with its content zones* to read module dates. Fetch at `contentLinkDepth: 1` — at depth `0` the modules come back as unexpanded references with no `properties`, and your date computation silently collapses to just the page-object date.

```ts
// lib/cms/getPage.ts
import getAgilitySDK from "lib/cms/getAgilitySDK"
import { cacheConfig } from "lib/cms/cacheConfig"
import { PageRequestParams } from "@agility/content-fetch/dist/methods/getPage"

export const getPage = async (params: PageRequestParams) => {
	const agilitySDK = getAgilitySDK()

	// Tag matches what the publish webhook busts on a page publish.
	agilitySDK.config.fetchConfig = {
		next: {
			tags: [`agility-page-${params.pageID}-${params.languageCode || params.locale}`],
			revalidate: cacheConfig.cacheDuration,
		},
	}

	return await agilitySDK.getPage(params)
}
```

### Step 2: Parse dates as UTC, and retry transient failures

Agility returns ISO strings with no timezone (`"2026-07-08T23:06:56.673"`), which JS would read as *local* time. Treat them as UTC so results are deterministic. And because the fetch API occasionally returns a `408`, wrap every call in a small retry — the SDK logs the error and resolves to `undefined` rather than throwing, so retry on a null result as well as on a thrown error.

```ts
const toMillis = (m?: string | Date | null): number => {
	if (!m) return 0
	if (m instanceof Date) return isNaN(m.getTime()) ? 0 : m.getTime()
	const hasTz = /([zZ])|([+-]\d{2}:?\d{2})$/.test(m)
	const t = new Date(hasTz ? m : `${m}Z`).getTime()
	return isNaN(t) ? 0 : t
}

const withRetry = async <T>(fn: () => Promise<T>, attempts = 3): Promise<T | undefined> => {
	for (let i = 0; i < attempts; i++) {
		try {
			const r = await fn()
			if (r !== undefined && r !== null) return r
		} catch { /* transient — retry */ }
	}
	return undefined
}
```

> **This retry is not optional.** The sitemap route is prerendered via ISR, so the whole date computation runs during `next build`. Without guarding against a transient `undefined`, a single flaky API call throws and **fails the entire deploy**.

### Step 3: Build the path → lastmod map

The naive approach — fetch each dynamic URL's content item one at a time — means thousands of API calls per build. Instead, **group dynamic nodes by their template (`pageID`) and pull each content list in bulk**, then read every date out of it. A thousand calls become a dozen. Large lists still page at 250 items/request (the API max), so loop `skip` until you've read them all.

The map value is `string | undefined`: every visible path is a key, and the value is `undefined` when no reliable date could be found — that URL will be emitted with no `<lastmod>`.

```ts
// lib/cms/getSitemapLastModifiedMap.ts (abridged)
import "server-only"
import { getSitemapFlat } from "lib/cms/getSitemapFlat"
import { getContentItem } from "lib/cms/getContentItem"
import { getContentList } from "lib/cms/getContentList"
import { getPage } from "lib/cms/getPage"

const CONTENT_LIST_PAGE_SIZE = 250 // API max per request

export type SitemapLastModifiedMap = Record<string, string | undefined>

export const getSitemapLastModifiedMap = async ({ channelName, languageCode }) => {
	// Guard the top-level fetch too: a transient failure returns {} rather than throwing.
	const sitemap = await withRetry(() => getSitemapFlat({ channelName, languageCode }))
	if (!sitemap) return {}

	// Real, sitemap-visible URLs only.
	const entries = Object.entries(sitemap).filter(([, n]: any) =>
		!n.isFolder && !n.redirect && n.visible?.sitemap)

	const dynamic = entries.filter(([, n]: any) => typeof n.contentID === "number")
	const staticPages = entries.filter(([, n]: any) => typeof n.contentID !== "number")

	const result: SitemapLastModifiedMap = {}

	// --- Dynamic pages: bulk-load each template's list once ---
	const byTemplate = new Map<number, any[]>()
	for (const [, n] of dynamic as any) {
		byTemplate.set(n.pageID, [...(byTemplate.get(n.pageID) ?? []), n])
	}

	const contentModified = new Map<number, number>()
	await Promise.all([...byTemplate.values()].map(async (nodes) => {
		// A failure here must never reject the whole build.
		try {
			// Learn the list's referenceName from one representative item.
			const sample = await withRetry(() =>
				getContentItem({ contentID: nodes[0].contentID, languageCode }))
			const referenceName = sample?.properties?.referenceName
			if (!referenceName) return

			// No sort — we read every item into a map keyed by contentID, so order
			// is irrelevant and sorting only adds server-side load (and timeout risk).
			let skip = 0, total = Infinity
			while (skip < total) {
				const list = await withRetry(() =>
					getContentList({ referenceName, languageCode, take: CONTENT_LIST_PAGE_SIZE, skip }))
				if (!list?.items) break // give up this template's remaining pages; per-item fills gaps
				total = list.totalCount ?? list.items.length
				for (const item of list.items) {
					contentModified.set(item.contentID, toMillis(item.properties?.modified))
				}
				if (!list.items.length) break
				skip += CONTENT_LIST_PAGE_SIZE
			}
		} catch { /* fall back to per-item below */ }
	}))

	const missing: any[] = []
	for (const [path, n] of dynamic as any) {
		const ms = contentModified.get(n.contentID)
		if (ms) result[path] = new Date(ms).toISOString()
		else if (contentModified.has(n.contentID)) result[path] = undefined // seen, no usable date -> omit
		else missing.push([path, n]) // never seen -> fetch individually
	}
	// per-item fallback for anything the bulk pass missed
	for (const [path, n] of missing) {
		const item = await withRetry(() => getContentItem({ contentID: n.contentID, languageCode }))
		const ms = toMillis(item?.properties?.modified)
		result[path] = ms ? new Date(ms).toISOString() : undefined
	}

	// --- Static pages: max(page date, newest module date) ---
	for (const [path, n] of staticPages as any) {
		const page = await withRetry(() =>
			getPage({ pageID: n.pageID, languageCode, contentLinkDepth: 1 }))
		let latest = toMillis(page?.properties?.modified)
		for (const zone of Object.values(page?.zones ?? {}) as any[]) {
			for (const mod of zone ?? []) {
				const ms = toMillis(mod?.item?.properties?.modified)
				if (ms > latest) latest = ms
			}
		}
		result[path] = latest ? new Date(latest).toISOString() : undefined // omit if none
	}

	return result
}
```

> In production, run the static-page and per-item fallback lookups with **bounded concurrency** (e.g. 10 at a time) rather than a single unbounded `Promise.all`, so a cold build doesn't burst hundreds of simultaneous requests at the API.

### Step 4: Wire it into the sitemap route

Use Next.js's route-level `revalidate` so the whole sitemap regenerates at most once a day. Between rebuilds the response is served from cache — zero API calls. On a rebuild, the per-page and per-content cache tags mean most fetches are still warm.

**Emit only `<loc>` and `<lastmod>`.** Skip `changefreq` and `priority` — Google ignores both, and omit `<lastmod>` entirely when the date is unknown.

```ts
// app/sitemap.tsx
import { getSitemapLastModifiedMap } from "lib/cms/getSitemapLastModifiedMap"
import { getSiteUrl } from "lib/utils/getSiteUrl"
import { cacheConfig } from "lib/cms/cacheConfig"
import { MetadataRoute } from "next"

export const revalidate = cacheConfig.pathRevalidateDuration // e.g. 86400 (24h)

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
	const channelName = process.env.AGILITY_SITEMAP || "website"
	const languageCode = process.env.AGILITY_LOCALES || "en-ca"

	const map = await getSitemapLastModifiedMap({ channelName, languageCode })
	const baseUrl = getSiteUrl()

	return Object.entries(map).map(([path, lastModified]) => {
		const isHome = path === "/home" // home lives at "/home" but is served at root
		const url = isHome ? `${baseUrl}/` : `${baseUrl}${path}`
		// No date? Emit <loc> alone — an omitted lastmod beats a wrong one.
		return lastModified ? { url, lastModified: new Date(lastModified) } : { url }
	})
}
```

Make the domain environment-driven (`getSiteUrl()`) so preview deploys emit their own host and production stays on your canonical domain — a preview host in your production sitemap points crawlers at URLs that just redirect.

### Sitemap gotchas

- **Don't ship `lastModified: new Date()`.** It's the whole problem. If you can't compute a real date for a URL, omit `lastmod` for that entry.
- **Don't emit `changefreq` or `priority`.** Google ignores them; a blanket `"daily"` only contradicts your honest `lastmod`.
- **Freshness to the day is fine.** Resist the urge to bust the sitemap on every content publish — a daily rebuild is plenty for search engines and keeps API load flat.
- **Guard against transient failures.** Because the route prerenders at build, an unhandled fetch error fails the deploy. Retry, and fall back to omitting a date.
- **Filter out folders, redirects, and hidden nodes** (`isFolder`, `redirect`, `visible.sitemap === false`) so they never reach the output.

---

## Part 2 — IndexNow

IndexNow is a simple protocol: you own a key, host it at a known URL so engines can verify it's really you, and POST a list of changed URLs. One submission is shared across all participating engines.

### Which engines accept IndexNow (mid-2026)

| Engine / surface | IndexNow? |
| --- | --- |
| **Bing** | ✅ Yes |
| **Yandex, Seznam.cz, Naver** | ✅ Yes |
| **Microsoft Copilot** | ✅ Yes (via Bing's index) |
| **ChatGPT Search** | ➖ Indirectly (Bing for discovery, plus its own `OAI-SearchBot` crawl) |
| **Google Search / AI Overviews / Gemini** | ❌ No |
| **Perplexity** | ❌ No |

Google announced in November 2021 that it was *testing* IndexNow and has never adopted it — so this is primarily a **Bing** play. Because Bing's index feeds Copilot and other AI answer engines, "in Bing's index quickly" increasingly means "citable by AI quickly." Keep the accurate sitemap from Part 1 as your Google lever.

### Step 1: Generate a key

Any string of 8–128 characters from `a–z A–Z 0–9 -`. A UUID with the dashes stripped works well:

```bash
node -e "console.log(require('crypto').randomUUID().replace(/-/g,''))"
```

Store it as an environment variable — set it in your host (e.g. Vercel, Production) and in `.env.local`:

```
INDEXNOW_KEY=your-generated-key
```

### Step 2: Serve the key file at the site root

Engines verify ownership by fetching `https://yourdomain.com/<key>.txt` and checking the body equals the key. It **must** be at the root — the key file's location scopes which URLs you're allowed to submit, so a nested path would only cover nested URLs.

If your app uses a catch-all route (`app/[...slug]`), a `/<key>.txt` request would fall into it. Serve it from middleware instead, before any host redirects:

```ts
// middleware.ts (near the top of the middleware function)
const indexNowKey = process.env.INDEXNOW_KEY
if (indexNowKey && request.nextUrl.pathname === `/${indexNowKey}.txt`) {
	return new NextResponse(indexNowKey, {
		status: 200,
		headers: {
			"Content-Type": "text/plain; charset=utf-8",
			"Cache-Control": "public, max-age=86400",
		},
	})
}
```

This is env-driven — nothing to commit, and rotating the key is just an env change.

### Step 3: Add a submit helper

```ts
// lib/indexnow/submitToIndexNow.ts
import { getSiteUrl } from "lib/utils/getSiteUrl"

const INDEXNOW_ENDPOINT = "https://api.indexnow.org/indexnow" // shared across engines

// Only ping from real production. INDEXNOW_ALLOW_NON_PROD=true to test.
const enabled = () =>
	process.env.VERCEL_ENV === "production" || process.env.INDEXNOW_ALLOW_NON_PROD === "true"

export const submitToIndexNow = async (urls: string | string[]) => {
	const key = process.env.INDEXNOW_KEY
	if (!key || !enabled()) return

	const baseUrl = getSiteUrl()
	const host = new URL(baseUrl).host

	const urlList = Array.from(new Set(
		(Array.isArray(urls) ? urls : [urls])
			.map((u) => { try { return new URL(u, `${baseUrl}/`).toString() } catch { return null } })
			.filter((u): u is string => !!u && new URL(u).host === host)
	)).slice(0, 10000) // IndexNow accepts up to 10k URLs per request

	if (!urlList.length) return

	try {
		const res = await fetch(INDEXNOW_ENDPOINT, {
			method: "POST",
			headers: { "Content-Type": "application/json; charset=utf-8" },
			body: JSON.stringify({ host, key, keyLocation: `${baseUrl}/${key}.txt`, urlList }),
			cache: "no-store",
			// Best-effort: a 10s timeout so a hung endpoint can't stall the webhook.
			signal: AbortSignal.timeout(10000),
		})
		if (!res.ok) console.error(`IndexNow: ${res.status} ${res.statusText}`)
	} catch (e) {
		console.error("IndexNow error:", e)
	}
}
```

Gating to production matters: preview and local deploys can't serve the key file at your canonical host, so their submissions would fail verification anyway.

### Step 4: Fire it from the publish webhook

Your revalidate webhook already knows the path that changed and is the natural place to hang this — the same publish event that clears caches and updates on-site search also pings IndexNow. Call it for both page and dynamic-page changes. It's `await`ed, but the internal timeout + swallowed errors mean it can't stall or fail the webhook — cache invalidation always takes priority.

```ts
// app/api/revalidate/route.ts (inside each change branch, after you resolve the path)
const publicPath = path === "/home" ? "/" : path
await submitToIndexNow(publicPath)
```

Expected responses: `200` (received and key already validated) or `202` (received, key validation pending). Both are success.

---

## Verify

**Sitemap:**

```bash
curl -s https://yourdomain.com/sitemap.xml | grep -A1 "<loc>" | head
```

Confirm `lastmod` values are varied and realistic — not all identical, not all "now". Spot-check a page you recently edited against a page you haven't touched in a year. There should be no `<changefreq>` or `<priority>` tags.

**IndexNow key file:**

```bash
curl -i https://yourdomain.com/<key>.txt
# Expect: 200, Content-Type text/plain, body == your key
```

**IndexNow submission:** publish a page and check your function logs for a successful (`200`/`202`) POST to `api.indexnow.org`. A `403` means the key file isn't verifying; a `429` means you're submitting too often.

> **One-time backfill:** since IndexNow only fires on *future* publishes, you can submit your genuinely-recently-changed URLs once (pull them from your sitemap's newest `lastmod` values). Submit real recent changes only — blasting your whole sitemap of unchanged URLs is against IndexNow etiquette.

---

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| Build fails prerendering `/sitemap.xml` | A transient API error (e.g. `408`) left a fetch `undefined` and threw | Wrap every fetch in `withRetry` and guard the result; never let one call fail the build. |
| All `lastmod` identical / "now" | Still emitting `new Date()`, or dynamic dates aren't resolving | Confirm the map builder is wired in and content lists return `properties.modified`. |
| Static page date too old | `getPage` called at `contentLinkDepth: 0` | Use depth `1` so module content items expand and expose their dates. |
| Sitemap build hammers the API | Per-URL fetching instead of bulk lists | Group dynamic nodes by template and pull each content list once; cache and bound concurrency. |
| `/<key>.txt` returns 404 | Env var added after the server started, or middleware matcher excludes `.txt` | Restart/redeploy; ensure the middleware `matcher` doesn't skip your key path. |
| IndexNow POST returns 403 | Key file not reachable at the canonical host | Deploy the middleware change and confirm the key file loads at your production domain. |

---

## Reference

- Google Search Central — [Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) and [Manage your crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget)
- [IndexNow documentation](https://www.indexnow.org/documentation)
- Next.js — [`sitemap.ts` / MetadataRoute.Sitemap](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap)

### Code paths in this implementation

- `lib/cms/getPage.ts` — page + zones at `contentLinkDepth: 1`
- `lib/cms/getSitemapLastModifiedMap.ts` — builds the path → lastmod map
- `lib/indexnow/submitToIndexNow.ts` — IndexNow submission helper
- `lib/utils/getSiteUrl.ts` — env-driven canonical base URL
- `app/sitemap.tsx` — the sitemap route
- `middleware.ts` — serves the IndexNow key file
- `app/api/revalidate/route.ts` — publish webhook that fires IndexNow