Before You Switch On AI Search, Test Retrieval on Your Own Knowledge Base

[Sreeram Nair]

Why I wrote this

A better reranker helps, but it can only rank what your knowledge base gives it. The way to know whether AI Search will work for your users is to test it on your own content, with your own questions, as your own users.

The recent How ServiceNow and NVIDIA Adapt Nemotron Rerank for Enterprise AI article is a great example of measuring retrieval properly. The team held the first stage fixed, scored each workload separately, and reported where the new model didn't win.

One row in their results stood out to me. On internal search mostly outside the fine-tuning domain, scores stayed around 0.40 (binary NDCG), well below in-domain search at 0.62. For customers, that's the row that matters, because every organisation's knowledge base is "out of domain" in some way: its own terms, its own structure, its own gaps.

This post is a practical way for platform teams to run a small version of that evaluation on their own instance, before AI Search goes live.

What a model can't fix

Most poor search results I've seen come from the content and its setup, not the ranking model. These are worth checking before you test anything:

• Duplicates and near-duplicates. Three versions of the same VPN article split the relevance between them, and the stale one often wins.
• Outdated articles. Retired articles that were never retired properly still rank, and AI answers built on them are confidently wrong.
• Answers hidden in attachments. If the real content is in a PDF or screenshot, check what your search configuration actually indexes.
• Titles that don't match how people ask. "KB0012345 - Procedure v3 FINAL" won't match "how do I reset my password".
• Access settings. User Criteria that are too broad can surface content people shouldn't see. Too narrow, and the right answer is invisible to the people who need it.

A reranker reorders what it's given. It can't add a missing answer or remove one a user shouldn't see.

Step 1: Build a test set from real questions

You need 50 to 100 real questions, each paired with the article or articles that correctly answer it. Real questions matter more than clever ones, because users rarely phrase things the way article authors do.

Good places to find them:

1. Search logs. The queries people actually typed into your portal or global search, especially frequent ones and ones that led to an incident anyway.
2. Incident short descriptions. Tickets resolved by pointing to a KB article give you the question and the right answer in one place.
3. Virtual Agent or chat transcripts. These capture how people ask in full sentences.
4. Your service desk team. Ask them for the ten questions they answer most often, in the caller's words.

For each question, have someone who knows the content mark which articles are relevant. Keep it simple: relevant or not. Include a handful of questions with no good answer in the knowledge base too, so you can see what search returns when it shouldn't return anything.

Step 2: Score it simply

Two numbers will tell you most of what you need: how often the right article is the top result, and how often it's in the top three. Both fit in a spreadsheet.

Measure What it tells you How to work it out

Top 1 rate	How often an AI answer built on the first result would be grounded in the right content	Share of questions where result 1 is relevant
Top 3 rate	How often the answer is at least close at hand	Share of questions with a relevant article in results 1 to 3
Misses	Where search fails completely	Questions with no relevant article in the top 10, read one by one
False confidence	What happens when there's no good answer	For no-answer questions, whether search or AI returns something confident anyway

The Nemotron article uses binary NDCG, which rewards relevant results more the higher they appear. It's a better single number if you're comparing configurations, but top 1 and top 3 are easier to explain to stakeholders.

The misses are worth more than the scores. Read every one and note why it failed: no article exists, the article exists but is badly titled, a duplicate outranked it, or the user couldn't see it.

Step 3: Test as real users, not as admin

Run the test set while impersonating real personas, because an admin sees everything and will get results no real user ever sees.

Pick three or four test users that represent your main groups, for example a general employee, a service desk agent, a manager and someone in a restricted area such as HR or Legal. Then run two checks:

• Can they find what they should? Run the normal test set as each persona. If the top 1 rate drops sharply for one group, the User Criteria on the right articles may be too narrow.
• Can they find what they shouldn't? Write a handful of questions that only restricted content can answer, such as salary bands or an ongoing investigation process, and run them as a general employee. Neither the search results nor the AI-generated answer should reveal that content.

The second check is the one to show your security team. A leak through an AI summary is just as serious as one through a search result, and it's easier to miss.

Step 4: Test the languages your users actually write in

If your users ask questions in more than one language, include those questions in the test set and score them separately. A good overall number can hide a poor result for a smaller group.

The Nemotron article showed multilingual retrieval improving from 0.7089 to 0.7513 after fine-tuning, but public multilingual benchmarks cover mostly widely used languages. Your users may write in languages with far less training data.

In New Zealand, for example, many government agencies publish content in te reo Māori, and people increasingly search that way too. It's worth testing questions in te reo Māori, English and a mix of both, because mixed-language queries are common and easy to forget. The same applies to any organisation with a significant community language.

Also check the reverse case: a question in one language where the only good answer is written in another. Whether that should work depends on your configuration, so it's worth knowing either way.

Step 5: Fix the content, then run it again

Use the misses from Step 2 as a work list, fix the content, and re-run the same test set. If the numbers move, you've proven the fix worked. If they don't, you've learned something too.

The usual fixes, roughly in order of effort:

1. Retire duplicates and outdated articles properly.
2. Rewrite titles and the first lines of key articles in the words people actually use.
3. Move answers out of attachments and into the article body.
4. Correct User Criteria that are too broad or too narrow.
5. Write the missing articles for the questions with no good answer.

Keep the test set and re-run it after every major change, whether that's a content clean-up, a search configuration change or a platform upgrade with a new model. Holding the questions fixed is what makes the before and after comparable, which is the same principle the Nemotron team used when they held the first-stage retrieval fixed.

A starter results sheet

One row per question per persona is enough, and you can paste this straight into a spreadsheet.

question,language,persona,relevant_articles,top_result,rank_of_first_relevant,should_be_hidden,leaked,miss_reason
How do I reset my password,en,Employee,KB0010001,KB0010001,1,FALSE,FALSE,
What are the salary bands for my role,en,Employee,KB0020001,KB0010450,,TRUE,FALSE,
Me pēhea au e tono whakatā ai,mi,Employee,KB0010300,KB0010877,4,FALSE,FALSE,Title doesn't match query

The rows above are placeholders to show the format, not real results.

Over to you

You don't need a research team to do this. Fifty real questions, three or four personas and a spreadsheet will tell you more about how AI Search will behave for your users than any benchmark can.

If you try it, I'd love to hear what you find. Were most of your misses down to content, configuration or the model? And has anyone tested te reo Māori or other community languages on their instance? Share in the comments, even if your numbers are small.