Your sitemap has been lying to Google (ours was too)

Our sitemap was stamping lastmod with the current time on every URL, every time it was generated. Twelve hundred pages, all shouting "I changed just now" at Google on every crawl. That is not a small cosmetic bug. A sitemap that lies about freshness is worse than one that says nothing at all, because you are actively teaching Google to stop trusting the one signal you most want it to trust.

We fixed it so every URL reports the date its content actually changed. While we were in there we wired up IndexNow, so Bing hears about a change the moment we publish instead of whenever a crawler wanders by.

This is a story about the two halves of getting indexed, the pull and the push, and why you want both.

The bug that looked like a feature

If you had opened our old sitemap code, you would have seen something that looks completely reasonable:

lastModified: new Date(),

Every entry, the current timestamp. It even feels right: yes, this is the sitemap, it is fresh, here is now. I suspect a lot of sitemaps out there have exactly this line in them, and everybody who wrote it thought they were doing the responsible thing.

They were not. Neither were we.

<lastmod> is not "when did you generate this sitemap." It is "when did this specific page last meaningfully change." Those are wildly different questions, and answering the first one when Google asked the second is how you end up with 1,200 URLs that all claim to have changed at 3:47 this afternoon.

Google is not new at this. It compares what your sitemap says against what it actually finds when it crawls. When your sitemap says everything changed but nothing did, it quietly files your lastmod under "ignore this site's dates" and moves on. The discount applies site-wide, not just to the pages that lied.

So the real cost was not a wrong date. It was that we had trained the most important reader we have to skip the field entirely.

Why Google cares about a date at all

Google does not recrawl your whole site every day. It cannot, and it would not want to, because crawling costs it money and costs you server load. Instead it spends a crawl budget: a rough per-site allowance of how much it will fetch and how often.

For a fifty-page brochure site, none of this is measurable. For a site with a thousand-plus URLs, a blog with years of archives, a resource library, a pile of dynamic detail pages, it is the whole game. If Google spends its budget re-fetching pages that have not changed since 2019, it has less left over for the post you shipped this morning.

An honest lastmod is how you spend that budget well. It says: these three pages changed this week, the other nine hundred are exactly as you last saw them. Google gets to skip the unchanged nine hundred and go straight to what is new. Your fresh content gets picked up faster, your stale content stops eating attention, and Google starts trusting your sitemap, which makes every future signal you send land harder.

This is the same principle behind most of the SEO best practices for a headless CMS: make the machine-readable layer of your site tell the truth cheaply, and the rest gets easier.

But there is a catch, and it is the interesting part.

"When did this page change" is a harder question than it sounds

What does it even mean for a page to have changed?

For a blog post or any dynamic detail page, it is easy. There is one content item behind that URL and it has a modified date. Done. The page changed when the content changed.

For a normal landing page, it is genuinely ambiguous. Our home page is not one thing. It is a page object with a stack of modules on it: a hero, a logo strip, a few feature sections, a testimonial. Each of those is its own piece of content with its own modified date. So when did the home page change? The page object itself might not have been touched in months while someone swapped out the hero this morning.

If you only look at the page object's date, which is the easy thing to do, you will report June 4th while the hero everyone actually sees was rewritten on July 8th. You would be lying to Google again, just more subtly.

So the rule we landed on: a dynamic page's date is its content item's date, and a normal page's date is the most recent thing on it, the page object or any module on it, whichever changed last. That is the date a human would give you if you asked "when did this page last change," and it is the date Google should get.

Getting that right meant actually looking at the content behind every URL, which brings us to the thing that nearly killed the whole idea.

Do not take down your own API to build a sitemap

The flat sitemap Agility hands you is just a list of URLs. It does not carry modified dates, so to build honest ones you have to go fetch the content behind each URL. Do that naively and you have a sitemap route that fires off a thousand-plus API calls every time a crawler asks for /sitemap.xml. Congratulations, you have built a very thorough way to DDoS yourself.

Two things save you. First, most of those thousand dynamic URLs are not a thousand different things. They are a handful of content lists (all the blog posts, all the starters, all the case studies) rendered through a few templates. So instead of fetching each post one at a time, you pull each list once, in bulk, and read every date out of it. A thousand calls collapse to a dozen.

Second, and this is the rule for anything that touches a content API on a hot path: cache it like you mean it. Every fetch rides a cache tag that only gets invalidated when that specific page or content item is republished, which is already how the rest of the site works, and the sitemap itself only regenerates once a day.

Steady state, it costs essentially nothing. A cold rebuild touches the API about a hundred times instead of a thousand, once, and then goes quiet. Freshness to the day is plenty for a sitemap. Nobody needs their lastmod accurate to the second, and pretending they do is how you end up back in trouble.

That is the Google half. An honest signal, cheaply produced, that you have earned the right to have believed. But it is still fundamentally a pull. You put the truth in a file and wait for Google to come read it. Which is a strange default when you think about it. You know the exact millisecond a page changed. Why are you waiting for someone to find out?

The other half: stop waiting, start telling

IndexNow is a protocol for pushing URL changes to search engines instead of waiting to be crawled. You generate a key, host it at a known URL so engines can verify it is really you, and POST a list of changed URLs the moment you publish. Participating engines pick it up, and because the network is shared, one ping reaches all of them.

You will notice Google is not one of them. Google announced in November 2021 that it was testing IndexNow and has never adopted it. Treat that as settled rather than imminent. For Google, the honest sitemap above is still your lever.

Which surfaces actually honour IndexNow

You will read that IndexNow is how you get cited by AI. Be skeptical, because a lot of that content is published by agencies selling AI-visibility services. Here is the actual coverage as of mid-2026.

Surface	Index behind it	Honours IndexNow	What actually moves the needle
Google Search, AI Overviews, Gemini	Google's own	No	Accurate sitemap lastmod, Search Console
Bing	Bing	Yes	IndexNow
Microsoft Copilot	Bing	Yes, via Bing	IndexNow
ChatGPT Search	Bing for URL discovery, plus its own OAI-SearchBot crawl	Indirectly	IndexNow, and do not block OAI-SearchBot
Perplexity	Its own index and crawler	No	Crawler access, sitemap
Yandex, Seznam, Naver	Own indexes	Yes	IndexNow

So the defensible version of the AI argument is narrow but real: Bing's index grounds Microsoft Copilot and still supplies URL discovery for ChatGPT Search, so a Bing-side ping does reach genuine AI surfaces. The overstated version is that this is how you win AI search. Google's AI Overviews and Gemini are the largest AI answer surface by query volume and are completely unaffected by IndexNow. Perplexity runs its own crawler.

The trend runs against the "everything is downstream of Bing" argument too. Microsoft retired the third-party Bing Search APIs on 11 August 2025 after several rounds of steep price increases, which pushed independent search products off Bing's data and onto crawlers of their own.

Which makes IndexNow a cost argument rather than a growth argument, and that is fine, because the cost is almost nothing. Roughly forty lines of code and one environment variable, once, in exchange for real coverage of Bing, Copilot, ChatGPT discovery and three regional engines. We hung it off the exact same publish webhook that already clears our caches and updates our search index, so the moment an editor hits Publish the page gets revalidated, re-indexed for on-site search, and pushed to IndexNow, all from one event. No new moving parts to babysit.

Just do not build a strategy on it. The accurate sitemap is the half that touches Google, and Google is still where the volume is.

Pull and push, doing different jobs

These two techniques are not competing. They cover for each other's weaknesses.

The sitemap lastmod is a pull optimization. It makes Google's own crawling efficient and trustworthy across your entire site, all the time, in the background. It is patient and comprehensive. What it is not is fast, because it works on Google's schedule rather than yours.

IndexNow is a push optimization. It is instant and targeted, this page right now, but it only reaches the engines that opted in and it is about individual changes rather than the shape of your whole site.

Put them together and the story is complete. Publish a post, and Bing and the AI surfaces that sit on it hear about it within minutes while Google finds it on its next efficient, well-signposted crawl. Nothing important waits longer than it has to, and nothing pointless gets crawled just to confirm it did not change.

We checked 28 other sitemaps before publishing this

Curiosity got the better of us, so we sampled the public sitemaps of 28 live sites on 28 July 2026: headless CMS vendors, hosting platforms, framework documentation, SEO tool companies and a few large publishers. Roughly 62,000 URLs in total. About one in five is sending Google a lastmod that is wrong rather than merely coarse.

Pattern	Sites	Who
The current time, regenerated on every request	2	Agility CMS (us, before this fix), Contentstack
Frozen at a single build timestamp	1	Payload
A large block clustered at build time	2	Wix, Hygraph
lastmod on almost nothing	1	Semrush, dates on 7% of URLs
No lastmod anywhere	5	Stripe, Webflow, Ahrefs, Directus, Astro
Roughly one real date per page	17	Moz, Smashing Magazine, Optimizely, Storyblok, Contentful, Netlify, Sanity, Vercel, Next.js, Kontent.ai, Prismic, Strapi, Ghost, WordPress.org, plus MDN, Squarespace and Sitecore using date-only values

We had the worst sitemap in our own sample

Fact-checking your own blog post and discovering you are the lead exhibit is a particular kind of humbling. Here is the same file, the same 1,028 URLs, before and after the fix described above:

Measure	Before	After
Distinct lastmod values	1	988
Share held by the most common value	100%	2.6%
Oldest date in the file	minutes ago	25 April 2020
URLs claiming to have changed in the last hour	1,028	0

The right-hand column is what a working implementation looks like: 988 distinct dates across 1,028 URLs, spread across six years, with the busiest single value holding 2.6% instead of everything. Google can now tell our pages apart.

The instructive ones

Payload is the cleanest specimen of the frozen-build-timestamp pattern. 443 URLs, exactly two distinct values: 2026-06-30T18:17:32.161Z and 2026-06-30T18:17:32.162Z. One millisecond apart. That is new Date() called inside a loop during a static build four weeks ago, and every page on the site has been reporting that same moment ever since.

Contentstack has the version most worth learning from, because it hides. Their 3,094 documentation URLs carry accurate, well-spread dates. Their 907 blog URLs, 170 resource URLs and 100 glossary URLs are all stamped with the current time. So 1,220 of 4,464 URLs are wrong while the rest are right, and the site-wide statistics look perfectly healthy: 1,550 distinct values across 4,464 URLs reads as a well-behaved sitemap. Two pipelines, two behaviours, one file. This is the same trap as the composed-page problem earlier in this post, just one level up.

Moz is the best in the sample: 1,502 distinct values across 1,511 URLs, with the most common value held by 0.1%. Effectively one real date per page. Smashing Magazine and Optimizely are close behind.

And then the five that omit lastmod entirely, including Stripe across 4,483 URLs and Webflow across 5,000. This is a defensible engineering choice rather than a failure. Absent data leaves the crawler on its own heuristics. Wrong data teaches it to distrust you. Google is explicit that it only uses lastmod when the values are consistently accurate, so Stripe omitting the field is a more correct position than Payload freezing it. The advice is not "always compute a lastmod." It is "compute it properly, or leave it out."

Ahrefs and Semrush both sell technical SEO tooling, and both landed in the bottom two rows. I mention it with no smugness whatsoever, because we had the single worst sitemap of the twenty-eight until about an hour before this post went live.

Three numbers that tell you which row you are in

1. Distinct lastmod values divided by total URLs. Close to 1.0 means roughly one real date per page. Close to zero means one date stamped everywhere.
2. The share held by the single most common value. Anything above about 20% deserves an explanation.
3. Whether the value changes between two requests seconds apart. If it does, you are calling new Date() at render time.

The third check is the one nobody runs, and it is the only one that catches the worst version of the bug. Two curl calls twenty seconds apart would have caught ours.

For composed sites, add a fourth: bucket the dates by URL path prefix, the way the Contentstack numbers above were split. A large, well-behaved section will happily mask a broken one in any site-wide average.

The takeaway

None of this is exotic. There is no growth hack here, no secret file, no acronym to buy a course about. It is two unglamorous pieces of plumbing done honestly: tell Google the truth about what changed, and tell everyone else the moment it does.

The thing worth internalizing is that indexing is a relationship, not a broadcast. A sitemap that cries wolf gets ignored. A ping that arrives the instant you publish gets rewarded. Both come down to respecting the reader on the other end, spending their crawl budget like it is real, because it is, and not wasting their time re-checking things that have not moved.

Do that and the index starts reflecting your site as it actually is, roughly as fast as it actually changes. Which, when you strip away all the noise, is the entire job.

Go check your own sitemap. If every URL has the same lastmod and it happens to be right about now, well. You know what to do. And if it turns out you are in that first row, you are in better company than you would think, including ours until this morning.

Frequently asked questions

What is lastmod in an XML sitemap?

<lastmod> is a per-URL field in an XML sitemap that reports when the content at that URL last meaningfully changed. It is not the time the sitemap file was generated. Crawlers use it to predict which URLs are worth re-fetching, so it is the cheapest way to aim a crawler at what actually moved.

Does Google actually use lastmod?

Google uses lastmod when the values are consistently accurate, and it verifies them against what it finds when it crawls. If your dates do not hold up, the signal gets discounted across the whole site. An omitted lastmod is better than a wrong one, because absent data leaves the crawler on its own heuristics while wrong data teaches it to distrust you.

How common is a broken sitemap lastmod?

In a sample of 28 live sites and roughly 62,000 URLs taken in July 2026, six were sending dates that were wrong rather than coarse and five omitted lastmod entirely, leaving seventeen that looked accurate. The most common failure is stamping the file generation time on every entry, which is easy to miss because the file looks perfectly correct when you open it in a browser.

Should a page's lastmod change when a module on it changes?

Yes. For a composed landing page, the honest date is the most recent of the page object's modified date and every content item rendered on it. Publishing a new hero into an untouched page really did change what a visitor sees, so it should bump the date. For a dynamic detail page there is only one content item behind the URL, so its modified date is the answer.

Does IndexNow help with Google or with AI search?

IndexNow does not reach Google at all, including AI Overviews and Gemini. It reaches Bing, Yandex, Seznam and Naver directly, and reaches Microsoft Copilot and ChatGPT URL discovery indirectly because both lean on Bing's index. Perplexity runs its own crawler and ignores it. Worth wiring up for the cost, not worth building a strategy on.

---

The full implementation, with the Next.js code for the lastmod map, the cached getPage wrapper, the middleware that serves the IndexNow key file, and the webhook wiring, is written up in Faster indexing with XML sitemaps and IndexNow in the Agility docs. If you are picking a platform and this kind of control matters to you, our take on the best CMS for SEO and on the Next.js 16.2 Adapter API cover the surrounding decisions.