# AI Search, Clean Markdown and an MCP Server in a Next.js Site

> Source: https://agilitycms.com/docs/nextjs/ai-search-markdown-and-mcp-endpoints

The demo site gives visitors an AI search box and an optional chat agent. It gives AI agents two more doors: a clean Markdown copy of every page and an MCP server. This article walks through all three in the code, so you can copy the parts you need.

The article covers the wiring in the Next.js app. For the ideas behind it, read [AI Search Explained](/docs/developers/ai-search-explained) and [Build a Chatbot or AI Agent on Your Agility Content](/docs/developers/build-a-chatbot-on-your-content). For the search index itself, see [Algolia](/docs/developers/algolia) and [Indexing Agility Content for Search](/docs/developers/indexing-content-for-search).

## See it in the demo

- AI search: open [demo.agilitycms.com](https://demo.agilitycms.com) and press Cmd+K (Ctrl+K on Windows and Linux), or click the sparkle button at the bottom right. The button only appears if the CMS setting "show AI search" is on.
- Clean Markdown: add `.md` to any page URL, for example [demo.agilitycms.com/about-us.md](https://demo.agilitycms.com/about-us.md). The home page is [/index.md](https://demo.agilitycms.com/index.md).
- Site index for agents: [demo.agilitycms.com/llms.txt](https://demo.agilitycms.com/llms.txt).
- MCP server: `https://demo.agilitycms.com/api/mcp` (see [Connect a client](#connect-a-client-to-the-mcp-server)).

The code is in the [demo repo](https://github.com/agility/nextjs-demo-site-2026). The files that matter:

| Part | Files |
| --- | --- |
| AI search and agent | `src/app/api/ai/search/route.ts`, `src/app/api/ai/agent/route.ts`, `src/lib/ai/`, `src/lib/cms-content/getAISearchConfig.ts`, `src/components/ai-search/`, `src/components/ai-agent/` |
| Clean Markdown | `src/proxy.ts`, `src/app/api/page-md/[...slug]/route.ts`, `src/lib/cms-content/pageToMarkdown.ts`, `src/app/llms.txt/route.ts` |
| MCP server | `src/app/api/mcp/route.ts` |

## Part 1: AI search and the chat agent

### The two endpoints

Both endpoints are `POST` routes that stream a response with the Vercel AI SDK (`ai` 5.x). Both set `maxDuration = 30`.

| Route | What it does |
| --- | --- |
| [`/api/ai/search`](https://github.com/agility/nextjs-demo-site-2026/blob/main/src/app/api/ai/search/route.ts) | One tool (`search`). The system prompt comes from the CMS. Allows up to five steps, so the model can search, read the results and then write an answer. |
| [`/api/ai/agent`](https://github.com/agility/nextjs-demo-site-2026/blob/main/src/app/api/ai/agent/route.ts) | Two tools (`search` and `contactCapture`). The system prompt is written in the route, not read from the CMS. |

Each route checks its configuration before it does anything. It returns a 500 if there are no AI provider variables or no Algolia variables, and a 503 if the CMS says AI search is switched off. The CMS switch controls both routes.

This is the core of the search route:

```ts
const result = streamText({
	model: model,
	system: systemPrompt,
	messages: convertToModelMessages(messages),
	tools: { search: searchTool },
	temperature: temperature,
	maxOutputTokens: maxOutputTokens,
	toolChoice: 'auto', // Let the model decide when to use tools
	stopWhen: stepCountIs(5), // Allow up to 5 steps for multi-step tool calls
})

return result.toUIMessageStreamResponse();
```

*From `src/app/api/ai/search/route.ts`.* `streamText`, `convertToModelMessages`, `stepCountIs` and `toUIMessageStreamResponse` are the AI SDK functions the routes use. `toUIMessageStreamResponse()` returns a streamed response that the browser's `useChat` hook reads.

### Choosing the model

`src/lib/ai/providers.ts` picks the provider from environment variables. Azure OpenAI wins if its variables are set. Otherwise it falls back to OpenAI.

| Provider | Variables it needs |
| --- | --- |
| Azure OpenAI | `AZURE_AI_RESOURCE`, `AZURE_AI_KEY`, `AZURE_AI_DEPLOYMENT` |
| OpenAI | `OPENAI_API_KEY`, `OPENAI_API_MODEL` |

The Algolia search tool also needs `ALGOLIA_APP_ID` and `ALGOLIA_SEARCH_API_KEY`. The code uses the search-only key, not a write key.

```ts
export function getAIModel(): LanguageModel | null {
  const { useAzureAI, useOpenAI } = validateAIEnvironment()

  if (useAzureAI && process.env.AZURE_AI_DEPLOYMENT && azure) {
    return azure(process.env.AZURE_AI_DEPLOYMENT)
  } else if (useOpenAI && openai && process.env.OPENAI_API_MODEL) {
    return openai(process.env.OPENAI_API_MODEL)
  }

  return null
}
```

*From `src/lib/ai/providers.ts`.* The providers come from `@ai-sdk/azure` and `@ai-sdk/openai`. To use another provider, change this one function.

### The search tool

`src/lib/ai/searchTool.ts` defines the tool the model calls. The model sends a `query` and an optional `limit` (default 10). The tool queries an Algolia index named `content` and returns `id`, `title`, `content`, `description`, `image` and `url` for each hit.

```ts
const response = await algolia.searchSingleIndex({
	indexName: 'content',
	searchParams
})
```

*From `src/lib/ai/searchTool.ts`.* The tool then tidies each hit: it strips a "Galaxy Tech" brand suffix from the title and builds an excerpt from `content` when a record has no `excerpt` or `description`. That cleanup is specific to this demo's brand. Replace it with your own.

The index name `content` is hard-coded in the tool, in `src/app/api/search/route.ts` and in the MCP server. Change it in all three places if your index has another name.

### Who fills the Algolia index

Not this repo. The demo only reads from Algolia. Nothing in the code writes records, and there is no Algolia write key, indexing script or reindex route. A comment in `src/lib/cms-content/resolveAgilityMetaData.ts` says that Algolia and Azure AI Search crawlers "still index these pages", which points to an external crawler over the rendered site, but the repo does not configure that crawler.

If you want to index straight from Agility instead, with a signed webhook and one record per section, follow [Indexing Agility Content for Search](/docs/developers/indexing-content-for-search) and [Algolia](/docs/developers/algolia). The tool above only needs records with a `title`, `url` and some text in `content`, `excerpt` or `description`.

### Settings held in the CMS

The search route reads its settings from a content list with the reference name `aisearchconfiguration` (the "AI Search Configuration" model). `getAISearchConfig` reads the first item for the current locale. The reference name is case-sensitive.

| CMS field | Used for |
| --- | --- |
| `showAISearch` | Turns the floating button and both API routes on or off. |
| `aISearchSummarySystemPrompt` | System prompt for `/api/ai/search`. |
| `maxTokens` | `maxOutputTokens` for both routes. |
| `temperature` | A dropdown. The code reads 0.1, 0.3, 0.5, 0.7 or 0.9 from the text. |
| `aiSearchHelp` | Placeholder text in the input. |
| `defaultPrompt1`, `defaultPrompt2`, `defaultPrompt3` | Suggested questions. Blank ones are dropped. |
| `fullAgentMode` | Opens the chat agent instead of the search modal. |
| `contactCaptureURL` | Where the agent's contact form posts. |

`getAISearchConfig` always returns something. If the item is missing or the fetch fails, it logs and returns defaults (a built-in system prompt, 2000 tokens, temperature 0.3, search on). If the item exists but `showAISearch` is empty, search is off.

Because the settings are content, an editor can change the prompt, the temperature or the suggested questions and publish, with no deploy.

### The user interface

The layout renders `FloatingAISearch` when `aiConfig.showAISearch` is true (`src/app/[locale]/layout.tsx`). It loads the modal with `next/dynamic` and `ssr: false`, so the AI libraries stay out of the first page load.

`AIModal` then picks one of two modals from `fullAgentMode`:

- `AISearchModal` (in `src/components/ai-search/`) calls `/api/ai/search`. When the model uses its search tool, the modal's `onToolCall` also runs a normal search against `/api/search` (through the `useSearch` hook) to show result cards next to the AI answer. `/api/search` is a plain Algolia query with no AI in it.
- `AIAgentModal` and `AIAgentChat` (in `src/components/ai-agent/`) call `/api/ai/agent`. Both use `useChat` from `@ai-sdk/react` with `DefaultChatTransport`. The chat renders `tool-search` results with `SearchResults` and `tool-contactCapture` results with `ContactCaptureForm`.

Components in `src/components/ai-elements/` (`conversation`, `message`, `prompt-input`, `response` and others) are the chat building blocks. The search and agent modals import only those four.

### The contact capture tool

When a visitor wants to get in touch, the agent can call `contactCapture`. The tool does not send anything. It returns the message and the `postURL` from the CMS, and the chat renders a form. The browser then posts the name, email and company to that URL as JSON (`ContactCaptureForm.tsx`). If `contactCaptureURL` is empty, the tool returns an error message and no form appears.

### Things to know before you copy it

- The agent route has no `stopWhen`. In AI SDK 5 that means a single step, so after a tool call the model does not get a second turn to write a summary. The search route sets `stepCountIs(5)`. Add the same line to the agent route if you want answers after searching.
- The agent route ignores the system prompt in the CMS and uses the one written in the route. Both routes call `getAISearchConfig({ locale: 'en-us' })`, so settings come from the English item.
- `submitContactData` in `contactCaptureTool.ts` is exported but nothing calls it. The form does its own `fetch`.
- The search tool logs every query to the server console.

## Part 2: Clean Markdown for agents

AI crawlers and agents read text better than a page full of navigation, scripts and layout. The demo serves a Markdown copy of every CMS page and an index of them.

### The `.md` twin

Any page path plus `.md` returns Markdown with `Content-Type: text/markdown`. Two files do the work.

First, `src/proxy.ts` catches the extension and rewrites the request:

```ts
} else if (ext === "md") {
	const mdPath = request.nextUrl.pathname.replace(/\.md$/, "")

	const homeTarget = await homePageRedirect(mdPath)
	if (homeTarget) {
		const url = request.nextUrl.clone()
		url.pathname = `${homeTarget === "/" ? "" : homeTarget}/index.md`
		return NextResponse.redirect(url, 308)
	}

	const mdUrl = new URL(`/api/page-md${mdPath}`, request.nextUrl.origin)
	return NextResponse.rewrite(mdUrl)
}
```

*From `src/proxy.ts`.* This branch sits before the branch for extension-less paths, because `.md` counts as an extension. The path travels in the URL, not in a query string, because the comment in the proxy says query strings added during a rewrite don't survive reliably. The home page is served at `/`, so its Markdown is at `/index.md`, and `/home.md` redirects there.

Second, `src/app/api/page-md/[...slug]/route.ts` does the work. It splits off a locale prefix (`/fr/about-us.md` is French), loads the page with the same `getAgilityPage` the HTML route uses, and converts it:

```ts
const markdown = await pageToMarkdown(agilityData)

return new Response(markdown, {
	headers: {
		"Content-Type": "text/markdown; charset=utf-8",
		"CDN-Cache-Control": "public, s-maxage=60, stale-while-revalidate=86400",
	},
})
```

*From `src/app/api/page-md/[...slug]/route.ts`.* A missing or unpublished page returns a plain 404.

Because it reads the same cached getters as the HTML pages, a `.md` response is invalidated by the same publish webhook and can't disagree with the page.

### How the Markdown is built

`pageToMarkdown` in `src/lib/cms-content/pageToMarkdown.ts` writes the page title as `# `, the SEO meta description as a quote, and then walks the page's content zones. For each component it fetches the content item and emits the text fields it recognises. Heading fields are `heading`, `title`, `subheading`, `eyebrow` and `name`. Body fields are `textblob`, `content`, `description`, `text`, `body`, `excerpt` and `summary`. HTML is converted with `htmlToMarkdown`.

The match is by field name, not by component. That is deliberate: components are defined in the CMS, so a hand-written mapping per component would silently skip any component you add later. The trade-off is that a component whose text lives in fields with other names, or in a list it references, contributes nothing. You can see this in the live `about-us.md`: some section headings appear with no body under them. If your components use other field names, extend those two lists.

### `llms.txt`

`src/app/llms.txt/route.ts` serves `/llms.txt`, following the [llmstxt.org](https://llmstxt.org) convention. It mixes two kinds of content:

- Hand-written text, kept in the route: what the site is, that the brand and prices are fictional, and the language setup.
- A page list generated from the cached flat sitemap (`getSitemapFlat`). It leaves out folders, redirects and pages hidden from the sitemap. A small `PAGE_PURPOSES` map adds a one-line description for a few anchor pages. Everything else is listed by title alone.

A generated list can't go stale when an editor adds a page. The hand-written part is the framing a generator can't write. The file points agents to the `.md` twins, and the response has the same `CDN-Cache-Control` header as the Markdown route. For more on the format, see [Making Your Agility-Powered Site Readable by AI](/docs/developers/making-your-site-readable-by-ai).

## Part 3: An MCP server over your pages

`src/app/api/mcp/route.ts` exposes the site to MCP clients such as Claude, Cursor or ChatGPT, so an agent can list, search and read pages without scraping. It uses the `mcp-handler` package (2.2.0) and exports one handler for `GET`, `POST` and `DELETE`:

```ts
const handler = createMcpHandler(
	(server) => {
		server.registerTool("list_pages", { /* ... */ }, async ({ locale }) => { /* ... */ })
		server.registerTool("get_page", { /* ... */ }, async ({ path }) => { /* ... */ })
		server.registerTool("search_content", { /* ... */ }, async ({ query, limit }) => { /* ... */ })
	},
	{
		serverInfo: { name: "galaxy-tech-demo-site", version: "1.0.0" },
	}
)

export { handler as GET, handler as POST, handler as DELETE }
```

*From `src/app/api/mcp/route.ts`, trimmed.*

### The tools

| Tool | Input | Returns |
| --- | --- | --- |
| `list_pages` | `locale` (optional, `en-us` or `fr`) | A text list of every published page with its title and full URL, from the flat sitemap. |
| `get_page` | `path`, such as `/about-us` or `/fr/blog/some-post` | The page as Markdown, built by the same `pageToMarkdown` as the `.md` twins. If there is no published page, it says so and points to `list_pages`. |
| `search_content` | `query`, `limit` (1 to 25, default 10) | Title, URL and summary for each hit from the same Algolia `content` index. If Algolia isn't configured it says so and suggests `list_pages` and `get_page`. |

The tools read through the same cached CMS getters as the site, with `preview: false`. An agent only sees what the site would serve.

One detail worth copying: this file imports `z` from `zod4`, not `zod`. The MCP SDK needs Zod 4, and the AI SDK in this project is pinned to Zod 3, so `package.json` aliases `zod4` to `npm:zod@^4` and only this file uses it.

### Connect a client to the MCP server

The endpoint speaks MCP over HTTP at `/api/mcp`. In an MCP client that supports remote servers, add the URL `https://demo.agilitycms.com/api/mcp`. To check it from a terminal, ask it for its tools:

```bash
curl -s -X POST https://demo.agilitycms.com/api/mcp \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```

The reply is a server-sent event whose data lists `list_pages`, `get_page` and `search_content`, each with its description and input schema. The server needs no key, because the content it serves is already public.

If you want the same pattern for your own content, see [Build an MCP Server over Your Agility Content](/docs/developers/build-an-mcp-server-for-your-content).

## Other API routes

Every route in `src/app/api`:

| Route | What it does |
| --- | --- |
| `/api/ai/search` | Streaming AI search with one `search` tool. See Part 1. |
| `/api/ai/agent` | Streaming chat agent with `search` and `contactCapture` tools. See Part 1. |
| `/api/search` | `POST { query, limit }`. A plain Algolia query on the `content` index, used for the result cards in the search modal. |
| `/api/mcp` | The MCP server. See Part 3. |
| `/api/page-md/[...slug]` | The Markdown twin of a page. Reached through the `.md` rewrite in `src/proxy.ts`, not called directly. |
| `/api/contact` | Validates required fields and the email format, then logs the submission to the server console and returns success. It has a `TODO` where real delivery goes, plus a fake one-second delay. It is a stub: it stores and sends nothing. |
| `/api/revalidate` | The publish webhook receiver. Clears cache tags (see below). |
| `/api/preview` and `/api/preview/exit` | Turn Next.js draft mode on and off for Agility preview. `/api/preview` validates the preview key with `validatePreview` first. |
| `/api/switch-locale` | Finds the same page in another locale from its `pageID` and redirects to it. |
| `/api/dynamic-redirect` | Resolves a `?ContentID=` link from the CMS to the dynamic page that renders that item. |

### What `/api/revalidate` clears

The route reads the webhook body (`state`, `languageCode`, `referenceName`, `contentID`, `pageID`) and acts on `Published`, `Deleted` and `Unpublished`. The tag names match the ones the getters set with `cacheTag`:

| Webhook carries | Tag revalidated | Set by |
| --- | --- | --- |
| `referenceName` | `agility-content-{referenceName lowercased}-{languageCode}` | `getContentList` |
| `contentID` | `agility-content-{contentID}-{languageCode}` | `getContentItem` |
| `pageID` | `agility-page-{pageID}-{languageCode}` | `getPage` |
| a page change, or a content change that backs a dynamic page | `agility-sitemap-flat-{languageCode}` and `agility-sitemap-nested-{languageCode}` | `getSitemapFlat`, `getSitemapNested` |

On a publish it also calls `revalidatePath` for the page's path when the sitemap shows one. A webhook with no `contentID` and no `pageID` (a URL redirection change) calls `BUILD_HOOK_URL` if set. A published `posts` item also starts the demo's social publishing. See [Caching with Next.js and Agility](/docs/nextjs/caching-with-next-js-and-agility) for how the tags fit together.

### Verify the webhook

Agility signs each webhook with its own per-webhook secret (tick **Enable secure delivery** on the webhook). A revalidation route should verify that signature and reject anything that fails, so only Agility can clear your caches or start a build. `AGILITY_SECURITY_KEY` is not involved: it validates preview keys, not webhooks. [Indexing Agility Content for Search](/docs/developers/indexing-content-for-search#receive-and-verify-the-webhook) shows how to verify the signature.

## Build it in your own site

1. Add the AI SDK packages (`ai`, `@ai-sdk/react`, and `@ai-sdk/azure` or `@ai-sdk/openai`) and copy `src/lib/ai/providers.ts`.
2. Get content into a search index and point `searchTool.ts` at it. Change the index name and the field names it reads.
3. Create an "AI Search Configuration" content model with the fields above and a list with the reference name `aisearchconfiguration`. Copy `getAISearchConfig.ts`.
4. Copy one of the two API routes. Add `stopWhen: stepCountIs(5)` if the model should answer after it searches.
5. Build the UI on `useChat` with `DefaultChatTransport` pointing at your route, or copy the components in `src/components/ai-search/`.
6. Add the `.md` branch to your proxy, the `page-md` route and `pageToMarkdown`. Check the heading and body field lists against your own components.
7. Add `llms.txt` and the MCP route. Replace the hand-written text with yours.
8. Verify webhook signatures before you turn on `/api/revalidate`.

## Related

- [Using the Advanced Next.js Demo Site](/docs/nextjs/using-the-advanced-next-js-demo-site)
- [Caching with Next.js and Agility](/docs/nextjs/caching-with-next-js-and-agility)
- [Algolia](/docs/developers/algolia)
- [AI Search Explained](/docs/developers/ai-search-explained)
- [Build a Chatbot or AI Agent on Your Agility Content](/docs/developers/build-a-chatbot-on-your-content)
- [Build an MCP Server over Your Agility Content](/docs/developers/build-an-mcp-server-for-your-content)
- [Making Your Agility-Powered Site Readable by AI](/docs/developers/making-your-site-readable-by-ai)

---

## Related pages

- [Using the Advanced Next.js Demo Site](https://agilitycms.com/docs/nextjs/using-the-advanced-next-js-demo-site.md)
- [Audience and Region Personalization with Next.js](https://agilitycms.com/docs/nextjs/audience-and-region-personalization-with-next-js.md)
- [Shared-Element Page Transitions with React ViewTransition](https://agilitycms.com/docs/nextjs/view-transitions-with-next-js.md)
- [A Provider-Agnostic Analytics Layer with PostHog in Next.js](https://agilitycms.com/docs/nextjs/analytics-layer-with-posthog-in-next-js.md)

Index of every docs page: https://agilitycms.com/docs/llms.txt
