Faster indexing with XML sitemaps and IndexNow

Two complementary techniques get your content discovered faster:

An honest sitemap.xml — every URL reports the date its content actually changed, so Google spends its crawl budget on what's new instead of re-checking what isn't.
IndexNow — a publish-time ping that tells Bing, Yandex, Seznam, Naver (and the AI search that rides on Bing's index) that a URL changed, without waiting for a crawl.

Think of them as pull and push. The sitemap makes Google's own crawling efficient and trustworthy across your whole site; IndexNow pushes individual changes out instantly to the engines that support it. You want both.

Why this matters

Google only trusts your sitemap dates if they're consistent with what it finds when it crawls. A sitemap that stamps every URL with the current time — the classic lastModified: new Date() bug — trains Google to ignore your lastmod entirely. An accurate one does the opposite: it earns the signal, and Google can skip the thousand unchanged URLs to spend its budget on the handful that actually changed.

Two things worth internalizing up front:

Google uses lastmod (when it's trustworthy) and nothing else from your sitemap. It ignores <changefreq> and <priority> — Google's own John Mueller and Gary Illyes have said so repeatedly. Don't emit them; a blanket changefreq of "daily" on every URL just contradicts the accurate lastmod you worked to produce.
No date is better than a wrong date. If you can't determine a reliable modified date for a URL, omit <lastmod> for that entry rather than inventing one.

Before you begin

You'll need:

An Agility CMS site on Next.js (App Router) using @agility/content-fetch.
The standard cached content wrappers most Agility starters ship with — getSitemapFlat, getContentItem, getContentList — each setting a Next.js cache tag and revalidate.
A publish webhook route (e.g. app/api/revalidate/route.ts) that Agility calls on publish. You'll extend it for IndexNow.

Part 1 — An accurate sitemap `lastmod`

Where the dates come from

The flat sitemap Agility returns is a dictionary of path → node. Each node has pageID, path, visible, redirect, isFolder, and — for dynamic pages — a contentID. What it does not contain is a modified date. So lastmod has to be assembled from the content behind each URL:

Dynamic pages (the node has a contentID): use the backing content item's properties.modified — the date the content itself changed.
Static pages (no contentID): use the most recent of the page object's properties.modified and every module/content item on the page. Publishing new content into a page should bump its date, even if the page object was untouched.

Step 1: A cached `getPage` wrapper

You need the page with its content zones to read module dates. Fetch at contentLinkDepth: 1 — at depth 0 the modules come back as unexpanded references with no properties, and your date computation silently collapses to just the page-object date.

// lib/cms/getPage.ts
import getAgilitySDK from "lib/cms/getAgilitySDK"
import { cacheConfig } from "lib/cms/cacheConfig"
import { PageRequestParams } from "@agility/content-fetch/dist/methods/getPage"

export const getPage = async (params: PageRequestParams) => {
	const agilitySDK = getAgilitySDK()

	// Tag matches what the publish webhook busts on a page publish.
	agilitySDK.config.fetchConfig = {
		next: {
			tags: [`agility-page-${params.pageID}-${params.languageCode || params.locale}`],
			revalidate: cacheConfig.cacheDuration,
		},
	}

	return await agilitySDK.getPage(params)
}

Step 2: Parse dates as UTC, and retry transient failures

Agility returns ISO strings with no timezone ("2026-07-08T23:06:56.673"), which JS would read as local time. Treat them as UTC so results are deterministic. And because the fetch API occasionally returns a 408, wrap every call in a small retry — the SDK logs the error and resolves to undefined rather than throwing, so retry on a null result as well as on a thrown error.

const toMillis = (m?: string | Date | null): number => {
	if (!m) return 0
	if (m instanceof Date) return isNaN(m.getTime()) ? 0 : m.getTime()
	const hasTz = /([zZ])|([+-]\d{2}:?\d{2})$/.test(m)
	const t = new Date(hasTz ? m : `${m}Z`).getTime()
	return isNaN(t) ? 0 : t
}

const withRetry = async <T>(fn: () => Promise<T>, attempts = 3): Promise<T | undefined> => {
	for (let i = 0; i < attempts; i++) {
		try {
			const r = await fn()
			if (r !== undefined && r !== null) return r
		} catch { /* transient — retry */ }
	}
	return undefined
}

This retry is not optional. The sitemap route is prerendered via ISR, so the whole date computation runs during next build. Without guarding against a transient undefined, a single flaky API call throws and fails the entire deploy.

Step 3: Build the path → lastmod map

The naive approach — fetch each dynamic URL's content item one at a time — means thousands of API calls per build. Instead, group dynamic nodes by their template (pageID) and pull each content list in bulk, then read every date out of it. A thousand calls become a dozen. Large lists still page at 250 items/request (the API max), so loop skip until you've read them all.

The map value is string | undefined: every visible path is a key, and the value is undefined when no reliable date could be found — that URL will be emitted with no <lastmod>.

// lib/cms/getSitemapLastModifiedMap.ts (abridged)
import "server-only"
import { getSitemapFlat } from "lib/cms/getSitemapFlat"
import { getContentItem } from "lib/cms/getContentItem"
import { getContentList } from "lib/cms/getContentList"
import { getPage } from "lib/cms/getPage"

const CONTENT_LIST_PAGE_SIZE = 250 // API max per request

export type SitemapLastModifiedMap = Record<string, string | undefined>

export const getSitemapLastModifiedMap = async ({ channelName, languageCode }) => {
	// Guard the top-level fetch too: a transient failure returns {} rather than throwing.
	const sitemap = await withRetry(() => getSitemapFlat({ channelName, languageCode }))
	if (!sitemap) return {}

	// Real, sitemap-visible URLs only.
	const entries = Object.entries(sitemap).filter(([, n]: any) =>
		!n.isFolder && !n.redirect && n.visible?.sitemap)

	const dynamic = entries.filter(([, n]: any) => typeof n.contentID === "number")
	const staticPages = entries.filter(([, n]: any) => typeof n.contentID !== "number")

	const result: SitemapLastModifiedMap = {}

	// --- Dynamic pages: bulk-load each template's list once ---
	const byTemplate = new Map<number, any[]>()
	for (const [, n] of dynamic as any) {
		byTemplate.set(n.pageID, [...(byTemplate.get(n.pageID) ?? []), n])
	}

	const contentModified = new Map<number, number>()
	await Promise.all([...byTemplate.values()].map(async (nodes) => {
		// A failure here must never reject the whole build.
		try {
			// Learn the list's referenceName from one representative item.
			const sample = await withRetry(() =>
				getContentItem({ contentID: nodes[0].contentID, languageCode }))
			const referenceName = sample?.properties?.referenceName
			if (!referenceName) return

			// No sort — we read every item into a map keyed by contentID, so order
			// is irrelevant and sorting only adds server-side load (and timeout risk).
			let skip = 0, total = Infinity
			while (skip < total) {
				const list = await withRetry(() =>
					getContentList({ referenceName, languageCode, take: CONTENT_LIST_PAGE_SIZE, skip }))
				if (!list?.items) break // give up this template's remaining pages; per-item fills gaps
				total = list.totalCount ?? list.items.length
				for (const item of list.items) {
					contentModified.set(item.contentID, toMillis(item.properties?.modified))
				}
				if (!list.items.length) break
				skip += CONTENT_LIST_PAGE_SIZE
			}
		} catch { /* fall back to per-item below */ }
	}))

	const missing: any[] = []
	for (const [path, n] of dynamic as any) {
		const ms = contentModified.get(n.contentID)
		if (ms) result[path] = new Date(ms).toISOString()
		else if (contentModified.has(n.contentID)) result[path] = undefined // seen, no usable date -> omit
		else missing.push([path, n]) // never seen -> fetch individually
	}
	// per-item fallback for anything the bulk pass missed
	for (const [path, n] of missing) {
		const item = await withRetry(() => getContentItem({ contentID: n.contentID, languageCode }))
		const ms = toMillis(item?.properties?.modified)
		result[path] = ms ? new Date(ms).toISOString() : undefined
	}

	// --- Static pages: max(page date, newest module date) ---
	for (const [path, n] of staticPages as any) {
		const page = await withRetry(() =>
			getPage({ pageID: n.pageID, languageCode, contentLinkDepth: 1 }))
		let latest = toMillis(page?.properties?.modified)
		for (const zone of Object.values(page?.zones ?? {}) as any[]) {
			for (const mod of zone ?? []) {
				const ms = toMillis(mod?.item?.properties?.modified)
				if (ms > latest) latest = ms
			}
		}
		result[path] = latest ? new Date(latest).toISOString() : undefined // omit if none
	}

	return result
}

In production, run the static-page and per-item fallback lookups with bounded concurrency (e.g. 10 at a time) rather than a single unbounded Promise.all, so a cold build doesn't burst hundreds of simultaneous requests at the API.

Step 4: Wire it into the sitemap route

Use Next.js's route-level revalidate so the whole sitemap regenerates at most once a day. Between rebuilds the response is served from cache — zero API calls. On a rebuild, the per-page and per-content cache tags mean most fetches are still warm.

Emit only <loc> and <lastmod>. Skip changefreq and priority — Google ignores both, and omit <lastmod> entirely when the date is unknown.

// app/sitemap.tsx
import { getSitemapLastModifiedMap } from "lib/cms/getSitemapLastModifiedMap"
import { getSiteUrl } from "lib/utils/getSiteUrl"
import { cacheConfig } from "lib/cms/cacheConfig"
import { MetadataRoute } from "next"

export const revalidate = cacheConfig.pathRevalidateDuration // e.g. 86400 (24h)

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
	const channelName = process.env.AGILITY_SITEMAP || "website"
	const languageCode = process.env.AGILITY_LOCALES || "en-ca"

	const map = await getSitemapLastModifiedMap({ channelName, languageCode })
	const baseUrl = getSiteUrl()

	return Object.entries(map).map(([path, lastModified]) => {
		const isHome = path === "/home" // home lives at "/home" but is served at root
		const url = isHome ? `${baseUrl}/` : `${baseUrl}${path}`
		// No date? Emit <loc> alone — an omitted lastmod beats a wrong one.
		return lastModified ? { url, lastModified: new Date(lastModified) } : { url }
	})
}

Make the domain environment-driven (getSiteUrl()) so preview deploys emit their own host and production stays on your canonical domain — a preview host in your production sitemap points crawlers at URLs that just redirect.

Sitemap gotchas

Don't ship lastModified: new Date(). It's the whole problem. If you can't compute a real date for a URL, omit lastmod for that entry.
Don't emit changefreq or priority. Google ignores them; a blanket "daily" only contradicts your honest lastmod.
Freshness to the day is fine. Resist the urge to bust the sitemap on every content publish — a daily rebuild is plenty for search engines and keeps API load flat.
Guard against transient failures. Because the route prerenders at build, an unhandled fetch error fails the deploy. Retry, and fall back to omitting a date.
Filter out folders, redirects, and hidden nodes (isFolder, redirect, visible.sitemap === false) so they never reach the output.

Part 2 — IndexNow

IndexNow is a simple protocol: you own a key, host it at a known URL so engines can verify it's really you, and POST a list of changed URLs. One submission is shared across all participating engines.

Which engines accept IndexNow (mid-2026)

Engine / surface	IndexNow?
Bing	✅ Yes
Yandex, Seznam.cz, Naver	✅ Yes
Microsoft Copilot	✅ Yes (via Bing's index)
ChatGPT Search	➖ Indirectly (Bing for discovery, plus its own `OAI-SearchBot` crawl)
Google Search / AI Overviews / Gemini	❌ No
Perplexity	❌ No

Google announced in November 2021 that it was testing IndexNow and has never adopted it — so this is primarily a Bing play. Because Bing's index feeds Copilot and other AI answer engines, "in Bing's index quickly" increasingly means "citable by AI quickly." Keep the accurate sitemap from Part 1 as your Google lever.

Step 1: Generate a key

Any string of 8–128 characters from a–z A–Z 0–9 -. A UUID with the dashes stripped works well:

node -e "console.log(require('crypto').randomUUID().replace(/-/g,''))"

Store it as an environment variable — set it in your host (e.g. Vercel, Production) and in .env.local:

INDEXNOW_KEY=your-generated-key

Step 2: Serve the key file at the site root

Engines verify ownership by fetching https://yourdomain.com/<key>.txt and checking the body equals the key. It must be at the root — the key file's location scopes which URLs you're allowed to submit, so a nested path would only cover nested URLs.

If your app uses a catch-all route (app/[...slug]), a /<key>.txt request would fall into it. Serve it from middleware instead, before any host redirects:

// middleware.ts (near the top of the middleware function)
const indexNowKey = process.env.INDEXNOW_KEY
if (indexNowKey && request.nextUrl.pathname === `/${indexNowKey}.txt`) {
	return new NextResponse(indexNowKey, {
		status: 200,
		headers: {
			"Content-Type": "text/plain; charset=utf-8",
			"Cache-Control": "public, max-age=86400",
		},
	})
}

This is env-driven — nothing to commit, and rotating the key is just an env change.

Step 3: Add a submit helper

// lib/indexnow/submitToIndexNow.ts
import { getSiteUrl } from "lib/utils/getSiteUrl"

const INDEXNOW_ENDPOINT = "https://api.indexnow.org/indexnow" // shared across engines

// Only ping from real production. INDEXNOW_ALLOW_NON_PROD=true to test.
const enabled = () =>
	process.env.VERCEL_ENV === "production" || process.env.INDEXNOW_ALLOW_NON_PROD === "true"

export const submitToIndexNow = async (urls: string | string[]) => {
	const key = process.env.INDEXNOW_KEY
	if (!key || !enabled()) return

	const baseUrl = getSiteUrl()
	const host = new URL(baseUrl).host

	const urlList = Array.from(new Set(
		(Array.isArray(urls) ? urls : [urls])
			.map((u) => { try { return new URL(u, `${baseUrl}/`).toString() } catch { return null } })
			.filter((u): u is string => !!u && new URL(u).host === host)
	)).slice(0, 10000) // IndexNow accepts up to 10k URLs per request

	if (!urlList.length) return

	try {
		const res = await fetch(INDEXNOW_ENDPOINT, {
			method: "POST",
			headers: { "Content-Type": "application/json; charset=utf-8" },
			body: JSON.stringify({ host, key, keyLocation: `${baseUrl}/${key}.txt`, urlList }),
			cache: "no-store",
			// Best-effort: a 10s timeout so a hung endpoint can't stall the webhook.
			signal: AbortSignal.timeout(10000),
		})
		if (!res.ok) console.error(`IndexNow: ${res.status} ${res.statusText}`)
	} catch (e) {
		console.error("IndexNow error:", e)
	}
}

Gating to production matters: preview and local deploys can't serve the key file at your canonical host, so their submissions would fail verification anyway.

Step 4: Fire it from the publish webhook

Your revalidate webhook already knows the path that changed and is the natural place to hang this — the same publish event that clears caches and updates on-site search also pings IndexNow. Call it for both page and dynamic-page changes. It's awaited, but the internal timeout + swallowed errors mean it can't stall or fail the webhook — cache invalidation always takes priority.

// app/api/revalidate/route.ts (inside each change branch, after you resolve the path)
const publicPath = path === "/home" ? "/" : path
await submitToIndexNow(publicPath)

Expected responses: 200 (received and key already validated) or 202 (received, key validation pending). Both are success.

Verify

Sitemap:

curl -s https://yourdomain.com/sitemap.xml | grep -A1 "<loc>" | head

Confirm lastmod values are varied and realistic — not all identical, not all "now". Spot-check a page you recently edited against a page you haven't touched in a year. There should be no <changefreq> or <priority> tags.

IndexNow key file:

curl -i https://yourdomain.com/<key>.txt
# Expect: 200, Content-Type text/plain, body == your key

IndexNow submission: publish a page and check your function logs for a successful (200/202) POST to api.indexnow.org. A 403 means the key file isn't verifying; a 429 means you're submitting too often.

One-time backfill: since IndexNow only fires on future publishes, you can submit your genuinely-recently-changed URLs once (pull them from your sitemap's newest lastmod values). Submit real recent changes only — blasting your whole sitemap of unchanged URLs is against IndexNow etiquette.

Troubleshooting

Symptom	Cause	Fix
Build fails prerendering `/sitemap.xml`	A transient API error (e.g. `408`) left a fetch `undefined` and threw	Wrap every fetch in `withRetry` and guard the result; never let one call fail the build.
All `lastmod` identical / "now"	Still emitting `new Date()`, or dynamic dates aren't resolving	Confirm the map builder is wired in and content lists return `properties.modified`.
Static page date too old	`getPage` called at `contentLinkDepth: 0`	Use depth `1` so module content items expand and expose their dates.
Sitemap build hammers the API	Per-URL fetching instead of bulk lists	Group dynamic nodes by template and pull each content list once; cache and bound concurrency.
`/<key>.txt` returns 404	Env var added after the server started, or middleware matcher excludes `.txt`	Restart/redeploy; ensure the middleware `matcher` doesn't skip your key path.
IndexNow POST returns 403	Key file not reachable at the canonical host	Deploy the middleware change and confirm the key file loads at your production domain.

Reference

Google Search Central — Build and submit a sitemap and Manage your crawl budget
IndexNow documentation
Next.js — sitemap.ts / MetadataRoute.Sitemap

Code paths in this implementation

lib/cms/getPage.ts — page + zones at contentLinkDepth: 1
lib/cms/getSitemapLastModifiedMap.ts — builds the path → lastmod map
lib/indexnow/submitToIndexNow.ts — IndexNow submission helper
lib/utils/getSiteUrl.ts — env-driven canonical base URL
app/sitemap.tsx — the sitemap route
middleware.ts — serves the IndexNow key file
app/api/revalidate/route.ts — publish webhook that fires IndexNow

Faster Indexing with XML Sitemaps and IndexNow

Why this matters

Before you begin

Part 1 — An accurate sitemap `lastmod`

Where the dates come from

Step 1: A cached `getPage` wrapper

Step 2: Parse dates as UTC, and retry transient failures

Step 3: Build the path → lastmod map

Step 4: Wire it into the sitemap route

Sitemap gotchas

Part 2 — IndexNow

Which engines accept IndexNow (mid-2026)

Step 1: Generate a key

Step 2: Serve the key file at the site root

Step 3: Add a submit helper

Step 4: Fire it from the publish webhook

Verify

Troubleshooting

Reference

Code paths in this implementation

Follow Us

Faster Indexing with XML Sitemaps and IndexNow

Why this matters

Before you begin

Part 1 — An accurate sitemap lastmod

Where the dates come from

Step 1: A cached getPage wrapper

Step 2: Parse dates as UTC, and retry transient failures

Step 3: Build the path → lastmod map

Step 4: Wire it into the sitemap route

Sitemap gotchas

Part 2 — IndexNow

Which engines accept IndexNow (mid-2026)

Step 1: Generate a key

Step 2: Serve the key file at the site root

Step 3: Add a submit helper

Step 4: Fire it from the publish webhook

Verify

Troubleshooting

Reference

Code paths in this implementation

Part 1 — An accurate sitemap `lastmod`

Step 1: A cached `getPage` wrapper