# Elastic > Source: https://agilitycms.com/docs/developers/elastic [Elasticsearch](https://www.elastic.co/elasticsearch) is the most widely deployed search engine, and it's the most flexible option in this set of guides: you control the mappings, the analyzers and every part of the query. It covers keyword search, vector search and **hybrid** search in one index, and it can also serve as the retrieval layer for an AI assistant. This guide builds on [Indexing Agility Content for Search](/docs/developers/indexing-content-for-search). Read that first. It covers the webhook, signature verification, re-fetching from the Fetch API, deletes and the full reindex, which are the same for every provider. This guide adds the Elastic-specific parts. > **Seen an older Agility + Elastic guide that uses App Search or the Elastic web crawler in Kibana?** Those products aren't part of Elasticsearch 9 or Elastic Cloud Serverless. This guide uses Elasticsearch's own APIs, which work everywhere. ## Choose where it runs - **Elastic Cloud Serverless — an "Elasticsearch" project** (recommended). There are no clusters to size, it's billed on usage, and semantic and hybrid search work out of the box. Note that it has a small baseline cost even when idle. - **Elastic Cloud Hosted** or **self-managed**. You get full control, but some features depend on your subscription: hybrid search with RRF, the Elastic Inference Service and Agent Builder need the **Enterprise** tier. Keyword search and everything in the first half of this guide work on any tier. ```bash npm install @elastic/elasticsearch ``` Version 9.5 of the client needs **Node.js 22 or later**, and it runs on the Node.js runtime, not an Edge runtime. ```bash # .env.local ELASTIC_URL=https://.es..elastic.cloud # from your project's connection details ELASTIC_INDEX=agility-content # an alias; see "Rebuilding" ELASTIC_WRITE_API_KEY=... # server only: webhook + reindex ELASTIC_SEARCH_API_KEY=... # server only: the search route ``` ## Create two API keys Create keys in Kibana (**Stack Management > API keys**) or with the API, each scoped to your index: ```ts // Read-only key for the search route await es.security.createApiKey({ name: "agility-search", role_descriptors: { search: { indices: [{ names: ["agility-content*"], privileges: ["read", "view_index_metadata"] }] }, }, }) // Write key for the webhook and reindex (manage is for creating indices and moving the alias) await es.security.createApiKey({ name: "agility-indexer", role_descriptors: { indexer: { indices: [{ names: ["agility-content*"], privileges: ["create_index", "index", "delete", "manage"] }] }, }, }) ``` Use the `encoded` value from each response. **Never send either key to the browser.** Serverless doesn't allow cross-origin requests anyway, so all searches go through a route on your server. ## Create the index The mapping mirrors the `SearchRecord` shape from the indexing guide. Create a versioned index, and point an alias at it, so that later you can rebuild without downtime: ```ts // scripts/create-index.ts import { Client } from "@elastic/elasticsearch" const es = new Client({ node: process.env.ELASTIC_URL!, auth: { apiKey: process.env.ELASTIC_WRITE_API_KEY! } }) const index = `agility-content-${Date.now()}` await es.indices.create({ index, mappings: { dynamic: false, properties: { kind: { type: "keyword" }, agilityId: { type: "integer" }, locale: { type: "keyword" }, referenceName: { type: "keyword" }, title: { type: "text", analyzer: "english" }, description: { type: "text", analyzer: "english" }, body: { type: "text", analyzer: "english" }, url: { type: "keyword", index: false }, updatedAt: { type: "date" }, }, }, }) await es.indices.putAlias({ index, name: "agility-content" }) ``` `dynamic: false` stops unexpected fields from being added to the mapping. The `english` analyzer handles stemming and stop words. For other locales, use one index per locale with the matching [language analyzer](https://www.elastic.co/docs/reference/text-analysis/analysis-lang-analyzer). ## Write to the index This is the `searchIndex` object the webhook route and the reindex script import: ```ts // lib/search/provider.ts import { Client } from "@elastic/elasticsearch" import type { SearchRecord } from "./types" const es = new Client({ node: process.env.ELASTIC_URL!, auth: { apiKey: process.env.ELASTIC_WRITE_API_KEY! }, serverMode: process.env.ELASTIC_SERVERLESS === "false" ? "stack" : "serverless", }) const index = process.env.ELASTIC_INDEX ?? "agility-content" export const searchIndex = { async upsert(record: SearchRecord) { await es.index({ index, id: record.id, document: record }) }, async upsertMany(records: SearchRecord[]) { if (!records.length) return const result = await es.helpers.bulk({ datasource: records, onDocument: (doc) => ({ index: { _index: index, _id: doc.id } }), }) if (result.failed) throw new Error(`${result.failed} of ${result.total} documents failed to index`) }, async remove(id: string) { await es.delete({ index, id }, { ignore: [404] }) }, } ``` - `index` with an explicit `id` is an upsert: it replaces the whole document. - `ignore: [404]` makes a delete of a document that's already gone succeed, so duplicate `Deleted` webhooks are harmless. - Don't pass `refresh: true` on webhook writes. Documents are searchable within about a second anyway, and forcing a refresh on every write slows the cluster down. Plug this into the route handler and reindex script from the [indexing guide](/docs/developers/indexing-content-for-search#the-endpoint), and publishing in Agility now updates Elasticsearch. ## Query from your site ```ts // app/api/search/route.ts import { Client } from "@elastic/elasticsearch" import type { SearchRecord } from "@/lib/search/types" const es = new Client({ node: process.env.ELASTIC_URL!, auth: { apiKey: process.env.ELASTIC_SEARCH_API_KEY! } }) export async function GET(req: Request) { const url = new URL(req.url) const q = (url.searchParams.get("q") ?? "").slice(0, 200) const locale = url.searchParams.get("locale") ?? "en-us" if (!q) return Response.json({ count: 0, hits: [] }) const res = await es.search({ index: process.env.ELASTIC_INDEX ?? "agility-content", size: 10, query: { bool: { must: [{ multi_match: { query: q, fields: ["title^3", "description^2", "body"], fuzziness: "AUTO" } }], filter: [{ term: { locale } }], }, }, highlight: { pre_tags: [""], post_tags: [""], fields: { body: { fragment_size: 160, number_of_fragments: 1 } }, }, _source: ["title", "description", "url", "kind"], }) const hits = res.hits.hits.map((h) => ({ id: h._id, ...h._source, caption: h.highlight?.body?.[0] })) const total = typeof res.hits.total === "number" ? res.hits.total : res.hits.total?.value return Response.json({ count: total, hits }, { headers: { "Cache-Control": "s-maxage=60, stale-while-revalidate=300" } }) } ``` `title^3` weights a title match three times as heavily as a body match, and `fuzziness: "AUTO"` tolerates small typos. The locale filter is applied on the server, so visitors can't remove it. ### Search UI The route returns `{ count, hits }`, with each hit carrying `title`, `url` and a highlighted `caption`. That's the shape the [minimal search box](/docs/developers/indexing-content-for-search#a-minimal-search-box) in the indexing guide expects. If you'd rather use a component library: - **[Search UI](https://www.elastic.co/docs/reference/search-ui)** — Elastic's React library. Use `@elastic/search-ui-elasticsearch-connector` through its **API proxy connector**, so the key stays on your server. Don't use the App Search connector; it's deprecated. - **[Searchkit](https://www.searchkit.co)** — a community project that lets Algolia's InstantSearch widgets run against Elasticsearch. ## Add semantic and hybrid search Semantic search matches on meaning, so "cancel my subscription" can find a page titled "Ending your plan". **Hybrid** search runs keyword and semantic retrieval together and merges the results, and it's usually better than either on its own. ![Your webhook route writes each record to the agility-content alias. Text fields use an English analyzer for keyword search and are copied into a semantic_text field, which Elasticsearch embeds and chunks itself. At query time, a keyword retriever and a semantic retriever run together and RRF fuses their results.](https://cdn.aglty.io/agility-cms-docs/images/developer/docs-diagram-elastic-hybrid.svg) With a `semantic_text` field, Elasticsearch generates the embeddings for you, **and** splits long text into passages automatically. There's no embedding code in your webhook. This works out of the box on Serverless and on Hosted with the Enterprise tier. 1. Find the embedding endpoint available to you, and **pin it**. The default for `semantic_text` has changed between versions, and an unpinned field can end up on a different model after an upgrade: ```bash GET _inference/_all ``` 2. Add a `semantic_text` field and copy the text fields into it. Put this in a **new** index, because the mapping of an existing field can't be changed: ```ts properties: { // …existing fields… title: { type: "text", analyzer: "english", copy_to: "semantic" }, description: { type: "text", analyzer: "english", copy_to: "semantic" }, body: { type: "text", analyzer: "english", copy_to: "semantic" }, semantic: { type: "semantic_text", inference_id: ".jina-embeddings-v5-text-small" }, // the ID from step 1 } ``` 3. Query with an RRF retriever, which searches the keyword fields and the semantic field and fuses the results: ```ts const res = await es.search({ index: process.env.ELASTIC_INDEX ?? "agility-content", size: 10, retriever: { rrf: { query: q, fields: ["title^3", "description^2", "body", "semantic"], filter: [{ term: { locale } }], rank_window_size: 50, }, }, highlight: { fields: { semantic: { type: "semantic", number_of_fragments: 1 } } }, _source: ["title", "description", "url", "kind"], }) ``` The `semantic` highlighter returns the passage that best matches the question, which makes a good result snippet. RRF results can't be sorted, and paging only reaches as far as `rank_window_size`. Things to know: - **Every index write re-embeds the document**, even if its text hasn't changed. That's fine at the rate editors publish, but a full reindex re-embeds everything, so budget for it. If you reindex often, store a hash of the text and skip unchanged items. - **Reranking** — for the highest relevance, wrap the retriever in a `text_similarity_reranker`, which re-scores the top results with a reranking model. See [semantic reranking](https://www.elastic.co/docs/solutions/search/ranking/semantic-reranking). - **Self-managed** clusters have no default embedding endpoint. Deploy a model to a machine learning node, or connect to the Elastic Inference Service, before `semantic_text` will work. ## Ground an AI assistant on your content [Agent Builder](https://www.elastic.co/docs/explore-analyze/ai-features/agent-builder/get-started) (GA) lets you build agents in Kibana that search your indices and answer questions from them. Every deployment on 9.2 or later, and every Serverless project, exposes those tools through an **MCP endpoint**: ``` {KIBANA_URL}/api/agent_builder/mcp ``` Point any MCP client (Claude, Cursor, VS Code and others) at it with an API key, and your AI assistant can search your Agility content directly. The older standalone `mcp-server-elasticsearch` package is deprecated in favour of this endpoint. On Serverless, the first 1,000 Agent Builder executions each month are free. On Hosted, Agent Builder needs the Enterprise tier. ## Rebuilding without downtime Because the app reads and writes through the `agility-content` alias, you can rebuild into a fresh index and switch over in one atomic step: ```ts const next = `agility-content-${Date.now()}` // 1. Create `next` with the mapping above // 2. Run the full reindex, writing to `next` // 3. Swap the alias and drop the old index const old = Object.keys(await es.indices.getAlias({ name: "agility-content" })) await es.indices.updateAliases({ actions: [ ...old.map((i) => ({ remove: { index: i, alias: "agility-content" } })), { add: { index: next, alias: "agility-content" } }, ], }) for (const i of old) await es.indices.delete({ index: i }) ``` Use the same approach to change a mapping, add the semantic field, or change the embedding model. ## OpenSearch [OpenSearch](https://opensearch.org) is an open-source fork of Elasticsearch 7.10. It's the usual choice on AWS (Amazon OpenSearch Service and OpenSearch Serverless). The overall integration is the same — the webhook, the record and the reindex — but the code in this guide won't run on it unchanged: - Use the `@opensearch-project/opensearch` client. The Elasticsearch client refuses to connect to non-Elastic servers. - Semantic and hybrid search use OpenSearch's own neural search, `hybrid` query and search pipelines rather than `semantic_text` and retrievers. You deploy and register the embedding model yourself. ## Troubleshooting - **`security_exception` on write.** The write key lacks `index` or `delete` on the index behind the alias. Check the `names` pattern covers `agility-content-*`. - **`version_conflict_engine_exception`.** Two deliveries for the same item raced each other. It's harmless because the next delivery or the reindex corrects it, but it's also a sign your endpoint is slow. Check the webhook's **History** for timeouts. - **Semantic queries fail with an inference error.** The `inference_id` doesn't exist on this deployment. Run `GET _inference/_all`. - **A result is stale.** Check the webhook's **History** in Agility, then run the reindex. See [Debugging](/docs/developers/indexing-content-for-search#debugging). ## Learn more - [Elasticsearch documentation](https://www.elastic.co/docs) - [Elasticsearch JavaScript client](https://www.elastic.co/docs/reference/elasticsearch/clients/javascript) - [Hybrid search](https://www.elastic.co/docs/solutions/search/hybrid-search) and [`semantic_text`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text) - [Elastic Cloud Serverless](https://www.elastic.co/docs/deploy-manage/deploy/elastic-cloud/serverless)