# AI Search Explained: Keyword, Vector, Hybrid and Semantic Ranking

> Source: https://agilitycms.com/docs/developers/ai-search-explained

"AI search" usually means several layers stacked on top of each other: keyword search, vector search, a way to merge the two, and a model that re-orders the top results. Each layer fixes a problem the one before it has, and each has its own way of failing.

This article explains what each layer does, in plain language. It then covers why to index sections instead of whole articles, why people and AI agents need different things from search, and what we measured when we rebuilt search for these docs. It ends with how to choose an approach, and how to test it on your own content before you commit.

If you are building a chatbot or giving AI agents access to your content, search is the part that decides whether the answers are right. See [Build a Chatbot or AI Agent on Your Agility Content](/docs/developers/build-a-chatbot-on-your-content) and [Build an MCP Server over Your Agility Content](/docs/developers/build-an-mcp-server-for-your-content) for what to build on top of it.

## The four layers

### Keyword search

Keyword search matches the words in a query against the words in your content. It ranks results by how often the query words appear in a record, weighted so that rare words count for more than common ones. BM25 is the most widely used ranking formula for this.

**What it's good at:** exact terms. API names, field names, error messages, product names and version numbers are matched exactly, and the results are fast, predictable and easy to explain. Engines built for search boxes, such as Algolia, also handle typos and half-typed words out of the box.

**Where it fails:**

- **Vocabulary mismatch.** If the reader uses a different word from the one your content uses, keyword search finds nothing. A reader asking about "languages" won't match pages that only ever say "locales".
- **Questions asked as sentences.** With Algolia's default settings, every word in the query must match. Agents and chatbots ask in full sentences, so on these docs a plain question about scheduling content returned 0 results, even though a page covers exactly that.

Keyword engines also rank differently from each other. Algolia ranks with a series of tie-breaking criteria rather than BM25. On our question-style test queries, BM25 in Azure AI Search ranked better (MRR 0.762 against 0.587, explained below).

### Vector search

Vector search compares meaning instead of words. An embedding model turns each piece of text into a long list of numbers (a vector), so that text with similar meaning gets similar vectors. At query time the question is turned into a vector with the same model, and the engine returns the records whose vectors are closest.

**What it's good at:** vocabulary mismatch and paraphrase. It connects "languages" to the pages about locales.

**Where it fails:**

- **Exact identifiers.** A vector represents what a passage is about, not the exact strings in it. A query that hinges on one API name or parameter can match a passage that is about the same topic but uses a different call. That is why vectors are normally combined with keyword search rather than used alone.
- **It always finds something.** Every query has a nearest neighbour, so vector search never returns zero results, even for a question your content can't answer. If you rely on zero-result searches to spot content gaps, you need a different signal once vectors are on.
- **Changing the model.** The index and the queries must use the same embedding model. If you change the model or its number of dimensions, you need a new index.

### Hybrid search

Hybrid search runs a keyword query and a vector query together and merges the two result lists into one. Keyword and vector scores are on different scales, so engines merge by rank position instead of raw score. Azure AI Search and Elastic both use Reciprocal Rank Fusion (RRF) for this: a record that ranks well in both lists ends up near the top.

**What it's good at:** getting the right page near the top. On these docs, hybrid search put the right page first for 40 of 51 questions and in the top five for 48, the best of everything we tested for questions asked in sentences.

**Where it fails:** the merged scores (around 0.03) are only good for ordering. They don't tell you how confident the engine is, so you can't use them to decide that a question has no answer.

### Semantic ranking (re-ranking)

A semantic ranker (also called a re-ranker) takes the top results from the first stage, reads each one together with the query, and scores how well that passage answers that question. It is slower than the first stage, so it only looks at a short list. Azure AI Search's semantic ranker re-scores the top 50 results and returns a score from 0 to 4. Elastic offers the same idea through its `text_similarity_reranker` retriever.

**What it's good at:** a confidence score on a fixed scale. You can use it to tell a chatbot when the best result is too weak to answer from. See [Decide when to say "I don't know"](/docs/developers/build-a-chatbot-on-your-content#decide-when-to-say-i-dont-know).

**Where it fails:** on our content it didn't improve the order. It put the right page first for 34 of 51 questions, against 40 for hybrid search alone. It tends to prefer pages that discuss a topic broadly over the page that answers the task. It can also only re-order what the first stage found, it adds latency and cost to every query, and on Azure's free plan it stops working (HTTP 402) once the monthly allowance is used. Measure it on your own content before you pay for it.

## Index sections, not whole articles

Every search engine returns **records**, and what you put in a record decides what a result looks like.

If each article is one record, a hit points at the whole article, and the snippet a reader sees may come from somewhere else on the page. Many setups also cap the body length to keep records small. On these docs a 5,000-character cap silently dropped the end of long articles, so text near the bottom of a long page was never indexed.

If each **section** is a record, the hit lands on the passage that answers the question. What worked on these docs:

- **One record per section, split at H2 and H3 headings.** The 451 articles in these docs become 4,890 sections.
- **A parent id on every section**, so you can show one result per page and delete every section of a page when it's unpublished.
- **A context prefix on each section** before it is embedded: `Title > Heading`, then the section text. A section called "Configuration" means little on its own. "Webhooks > Configuration" does.
- **Collapse results to one per page** when you show them, using the parent id.
- **Keep code blocks.** Developer questions are often answered by code. Stripping code makes snippets prettier but loses answers.
- **Don't cap the body.** Sections remove the need for a cap.
- **Normalize line endings before you split.** More than half of our articles were saved with Windows line endings (CRLF). In JavaScript, `.` in a regular expression doesn't match `\r`, so a heading pattern such as `^##\s+(.+)$` silently failed on every one of them. 211 articles were cut into blind fixed-length chunks with no headings, and 284 of 451 records in our live site search had no headings indexed. Replace `\r\n` with `\n` first, then read a few stored records to check the result.

**Splitting does not improve ranking by itself.** We expected it to, and measured it twice, the second time with the line-ending bug fixed: an Algolia index with one record per section ranked the same as one record per article (MRR 0.594 against 0.587, with 8 questions better and 9 worse). Index sections for the reasons that hold: the snippet is the passage that answers the question, a chatbot can quote it directly, long articles aren't cut off, and code stays in.

Splitting works best when each section makes sense on its own. [Write chunk-friendly content](/docs/developers/making-your-site-readable-by-ai#6-write-chunk-friendly-content) covers how. For how to keep sections in sync as editors publish, see [Indexing Agility Content for Search](/docs/developers/indexing-content-for-search#index-sections-not-whole-pages).

## People and AI agents search differently

The same index serves two very different kinds of searcher:

| | People in a search box | AI agents and chatbots |
| --- | --- | --- |
| Typical query | One to three words | A full question, often 6 to 13 words |
| Typos and half-typed words | Common | Rare |
| Searches per question | One per keystroke | One or a few |
| Latency | Every keystroke, so tens of milliseconds matter | Small next to the model's own time |
| What they need back | A list of pages to click | Passages to read and cite |

That difference showed up clearly in our tests. On short, real search-box queries (typos and half-typed words included), Algolia with its defaults was as good as the best Azure setup and about five times faster. On questions asked in sentences, Azure's hybrid search was far ahead. The Agility docs now use both: Algolia behind the search box and Azure AI Search behind the [Knowledgebase MCP server](/docs/overview/agility-knowledgebase-mcp-server) that AI agents call. [Search for People and for AI Agents](/docs/developers/search-for-people-and-ai-agents) shows how that setup works.

## What we measured on the Agility docs

In October 2026 we compared search approaches on these docs. It is one measurement, on one content set, at one point in time. Treat it as an example of the method and the size of the differences, not as a benchmark of any vendor.

### The setup

- **Content:** the Agility docs, 451 articles managed in Agility CMS, split into 4,890 sections. Both engines were filled from the CMS by the same code.
- **Algolia:** the docs site's existing index (one record per article), plus a second index with one record per section and the same settings, to measure splitting on its own.
- **Azure AI Search:** one record per section, code blocks kept. Keyword search used BM25 with the `en.microsoft` analyzer. Vectors came from `text-embedding-3-large` (3,072 dimensions) on Azure OpenAI, with an integrated vectorizer on the index so the search service embeds queries itself.
- **Agent test set:** 52 questions in the style agents ask them, taken from real customer searches, pages AI agents fetch, the most-read pages and agent-style phrasing, plus one the docs cannot answer.
- **Search-box test set:** 36 short queries copied as people typed them into the docs search box (typos and half-typed words included), plus a few cut off mid-word.
- **Unanswerable questions:** 9 questions the docs don't cover, each checked against the full text of every section.

For every set we wrote down the correct pages and committed them to git **before** scoring any engine.

### Questions asked in sentences (agents and chatbots)

51 answerable questions. Higher is better, except for the last column.

| Engine | Right page first | In top 3 | In top 5 | MRR@10 | Missed |
| --- | --- | --- | --- | --- | --- |
| Algolia, search-box settings | 13 | 16 | 18 | 0.291 | 33 |
| Algolia, settings for sentence-style questions | 25 | 34 | 37 | 0.587 | 13 |
| Algolia, one record per section (same settings) | 25 | 35 | 37 | 0.594 | 12 |
| Azure, keyword only | 32 | 45 | 46 | 0.762 | 3 |
| **Azure, keyword + vector (hybrid)** | **40** | **47** | **48** | **0.853** | **2** |
| Azure, hybrid + semantic ranker | 34 | 47 | 48 | 0.796 | 2 |

**MRR@10** (mean reciprocal rank) scores each question 1 if the right page is first, 1/2 if second, 1/3 if third, and so on, down to 0 if it isn't in the top 10, then averages over all questions. It rewards putting the right page near the top, not just finding it. **Missed** counts the questions where the right page was not among the top 10 distinct pages at all.

### Short search-box queries (people)

36 queries.

| Engine | Right page first | In top 3 | In top 5 | MRR@10 | Missed | Typical time |
| --- | --- | --- | --- | --- | --- | --- |
| **Algolia, search-box settings** | **30** | **34** | **34** | **0.886** | **0** | **~50 ms** |
| Azure, keyword only | 25 | 27 | 27 | 0.718 | 9 | ~60 ms |
| Azure, keyword with typo and prefix matching | 29 | 35 | 35 | 0.884 | 1 | ~70 ms |
| Azure, hybrid with typo and prefix matching | 31 | 34 | 36 | 0.911 | 0 | ~260 ms |

Times are from a test script to each service, so compare them with each other rather than read them as page load times. Azure matches whole words unless you ask otherwise. "Typo and prefix matching" means rewriting the query in Lucene syntax so each word may have one typo and the last word may be a prefix.

### Which differences are real

With 51 questions, one question moves MRR by about 0.02. So we compared engines question by question and computed a 95% interval for each difference:

| Comparison | Difference in MRR | 95% interval | Better / worse |
| --- | --- | --- | --- |
| Agents: Azure hybrid vs Algolia | +0.28 | +0.16 to +0.40 | 23 / 5 |
| Agents: adding vectors to keyword search | +0.09 | +0.03 to +0.16 | 13 / 1 |
| Agents: adding the semantic ranker to hybrid | −0.07 | −0.16 to +0.02 | 7 / 13 |
| Agents: Algolia sections vs whole articles | +0.00 | −0.08 to +0.09 | 8 / 9 |
| Search box: Azure with typo and prefix matching vs Algolia | −0.01 | −0.10 to +0.09 | 4 / 4 |

An interval that includes zero means we can't tell the two apart on this test set.

### What we learned

1. **For questions in sentences, hybrid search won clearly.** It found the right page first for 40 of 51 questions, against 25 for our tuned Algolia setup. Both the keyword ranking (BM25 weighs rare words) and the vectors contributed.
2. **For the search box, Algolia's defaults were already right.** Short, misspelled and half-typed queries are what it's built for. Azure needed query rewriting to catch up, and hybrid search was five times slower per keystroke.
3. **The semantic ranker didn't improve the order on our content.** Its value is the confidence score, not the ranking.
4. **Splitting into sections didn't change ranking.** Do it for snippets, passages and code, not for relevance.
5. **Agents and chatbots ask in sentences.** With Algolia's defaults every word must match, which returned 0 results for many natural-language questions. Four query settings fixed that for agent traffic: `queryLanguages: ["en"]`, `removeStopWords: true`, `ignorePlurals: true` and `removeWordsIfNoResults: "allOptional"`. The "sentence-style" Algolia rows use them.
6. **A confidence score helps a chatbot say "I don't know", but it isn't a clean cutoff.** On the semantic ranker's 0 to 4 scale, the 9 unanswerable questions scored between 1.96 and 2.62, and every answerable question's top result scored 2.28 or more. Refusing below 2.3 caught 7 of the 9 and wrongly refused 1 of the 51. Use the score alongside the model's own reading of the passage.
7. **Some misses are content gaps, not engine problems.** A question about letting editors click on the page to change it ranked poorly with every engine, because no page describes the feature in those words. Search analytics, especially zero-result and low-click queries, tell you which words your readers use that your content doesn't.
8. **Check that a full reindex sees everything.** Our full reindex read article lists through GraphQL without `take`. GraphQL lists default to 50 items, so three categories came back with exactly 50 articles and 89 articles were missing. Because the reindex replaces the whole index, it would have deleted them from live search. Pass an explicit `take`, and refuse to replace an index when a list comes back exactly at the limit.
9. **Read your index, not just your scores.** The line-ending bug above only showed up when we read the stored sections of one page. Every aggregate number looked plausible.

### Costs on Azure

- The Basic tier is the practical starting point. A single replica has no availability SLA for queries, so add a second replica if search is critical.
- The semantic ranker's free plan has a monthly allowance, after which semantic queries fail with HTTP 402 until you switch to the paid plan. If you leave it off, as our results suggest you can, this doesn't arise.
- Embedding all 4,890 sections took about two minutes and cost cents. A rebuild where nothing changed re-embeds nothing if you store a hash of each section's text.
- Hybrid search also calls the embedding model once per query, through the index's vectorizer.
- In production, prefer the search service's managed identity over an API key for the vectorizer's connection to Azure OpenAI.

Check current prices on the vendor's pricing page. They change.

## Choosing an approach

Every engine below is fed from Agility the same way: a webhook on publish, a re-fetch from the Fetch API, and a nightly reindex. [Indexing Agility Content for Search](/docs/developers/indexing-content-for-search) covers that part once for all of them. [Searching Content at Scale](/docs/overview/searching-content-at-scale) compares the providers side by side.

Start from who is searching:

- **People typing into a search box:** a keyword engine with typo tolerance and prefix matching, such as Algolia. It's fast, and it's what our search-box test favoured.
- **AI agents, MCP servers and chatbots:** hybrid search. Leave a semantic ranker off unless you need its score or it measurably helps on your content.
- **Both:** you can feed two engines from the same pipeline, as the Agility docs do. See [Search for People and for AI Agents](/docs/developers/search-for-people-and-ai-agents).

We only measured two of these providers, and only in the configurations described above. In particular, the Algolia rows don't include NeuralSearch, so they don't measure Algolia's own hybrid search.

- **[Algolia](/docs/developers/algolia)**: hosted keyword search with fast setup and ready-made UI libraries. Hybrid keyword and vector search is available as NeuralSearch on the Elevate plan. For agent and chatbot traffic, apply the four query settings from lesson 5 above.
- **[Azure AI Search](/docs/developers/azure-ai-search)**: keyword, vector, RRF fusion and the semantic ranker in one service. Its hybrid search scored best in our test for questions asked in sentences. A good fit if your organization already runs on Azure.
- **[Elastic](/docs/developers/elastic)**: a `semantic_text` field embeds and splits long text for you, an RRF retriever combines keyword and semantic search, and `text_similarity_reranker` adds re-ranking. Runs as a serverless project, on Elastic Cloud or self-managed. We did not measure it.
- **[Coveo](/docs/developers/coveo)**: enterprise relevance and personalization, with generative answering and a Passage Retrieval API that are licensed separately. We did not measure it.

Whichever you choose, index sections for the reasons in [Index sections, not whole articles](#index-sections-not-whole-articles).

## Measure before you choose

Search quality depends on your content and your readers' questions, so the only reliable comparison is one you run yourself. It takes a day or two.

1. **Collect real questions, one set per kind of searcher.** For agents, about 50 questions phrased as full sentences, from your search logs, the pages AI agents fetch most, your most-read pages and support tickets. For a search box, 30 or so short queries copied exactly as people typed them, typos included. Add several questions your content cannot answer.
2. **Write down the right answer first.** For each question, record the URL of the page that answers it (or several, if more than one is acceptable). Mark the unanswerable ones. Commit the file to git **before** you score anything, so you can't tune the answers to fit the results.
3. **Score each engine the same way.** For each question, find the rank of the first correct page in the top 10. Report how many questions had it first, in the top 3, in the top 5, and the MRR@10. For unanswerable questions, record the top score instead.
4. **Check which differences are real.** Count the questions each engine got better and worse, not just the averages. With 50 questions, a difference of one or two questions is noise.
5. **Call the engine directly, then test through the real interface.** Score engines through their APIs, with analytics turned off, so test traffic doesn't pollute your production search analytics. Then run the same questions through the actual product, such as your MCP server or chat route. Our MCP server refused queries longer than 12 words before searching, which silently dropped 2 of our 51 questions until we tested through it.
6. **Keep your test questions out of your content.** Once you publish what you learned, your examples may quote your test questions. Our own search articles did, and Algolia started ranking them first for those questions. Exclude such pages when scoring, or write fresh questions.
7. **Re-run after every change.** Keep the test sets in your repo and re-run them when you change the index, the query settings or the engine.

A scoring function is a few lines:

```ts
type Case = { question: string; expected: string[] } // expected page URLs; empty = unanswerable

// rankedUrls: the engine's results for each question, best first, collapsed to one per page
export function score(cases: Case[], rankedUrls: string[][]) {
  const answerable = cases.map((c, i) => ({ c, urls: rankedUrls[i] })).filter(({ c }) => c.expected.length > 0)
  const ranks = answerable.map(({ c, urls }) => {
    const i = urls.slice(0, 10).findIndex((u) => c.expected.includes(u))
    return i === -1 ? null : i + 1
  })
  const within = (k: number) => ranks.filter((r) => r !== null && r <= k).length
  return {
    questions: answerable.length,
    first: within(1),
    top3: within(3),
    top5: within(5),
    mrr10: ranks.reduce((sum: number, r) => sum + (r ? 1 / r : 0), 0) / answerable.length,
    missed: ranks.filter((r) => r === null).length,
  }
}
```

Keep the test sets once you have them. The same questions are how you evaluate a chatbot or an MCP server built on this search later.

## Related

- [Search for People and for AI Agents](/docs/developers/search-for-people-and-ai-agents)
- [Build a Chatbot or AI Agent on Your Agility Content](/docs/developers/build-a-chatbot-on-your-content)
- [Build an MCP Server over Your Agility Content](/docs/developers/build-an-mcp-server-for-your-content)
- [Indexing Agility Content for Search](/docs/developers/indexing-content-for-search)
- [Searching Content at Scale](/docs/overview/searching-content-at-scale)
- [Making Your Agility-Powered Site Readable by AI](/docs/developers/making-your-site-readable-by-ai)

---

## Related pages

- [Indexing Agility Content for Search](https://agilitycms.com/docs/developers/indexing-content-for-search.md)
- [Algolia](https://agilitycms.com/docs/developers/algolia.md)
- [Azure AI Search](https://agilitycms.com/docs/developers/azure-ai-search.md)
- [Coveo](https://agilitycms.com/docs/developers/coveo.md)
- [Elastic](https://agilitycms.com/docs/developers/elastic.md)
- [Build a Chatbot or AI Agent on Your Agility Content](https://agilitycms.com/docs/developers/build-a-chatbot-on-your-content.md)
- [Build an MCP Server over Your Agility Content](https://agilitycms.com/docs/developers/build-an-mcp-server-for-your-content.md)
- [Search for People and for AI Agents](https://agilitycms.com/docs/developers/search-for-people-and-ai-agents.md)

Index of every docs page: https://agilitycms.com/docs/llms.txt