ServiceNow AI Search returning irrelevant Knowledge Articles

[AftabS]

Hi everyone,

I am trying to implement semantic search using ServiceNow AI Search for Knowledge Articles.

So far, I have configured the following:

• Created a Search Profile

• Configured the required search source

• Added Synonyms

• Configured Result Improvement Rules

• Configure Search Application

• Tested the search using the API/payload

However, I am facing an issue with the search results.

When I send my search query using the payload, the response contains some irrelevant knowledge articles along with the relevant ones. I only want the results that are actually relevant to the user's search query.

For example, if I search for a specific topic, the response contains the expected KB articles, but it also returns other KB articles that are not related to the search.

I have attached screenshots showing:

1. My Search Profile configuration

2. Search Source configuration

3. Synonym configuration

4. Result Improvement Rule configuration

5. The request payload

6. The response showing the irrelevant KB articles

Could someone please help me understand:

• Why are these irrelevant Knowledge Articles being returned?

• Is there a way to configure AI Search so that only highly relevant KB articles are returned?

• Should I use Result Improvement Rules, Search Source conditions, or some other AI Search configuration to filter out irrelevant results?

• Is there a relevance/threshold setting that I can configure for the API response?

Any guidance on how to troubleshoot and improve the relevance of the returned KB articles would be greatly appreciated.

Thanks in advance!

[vaishali231]

Hey @AftabS 

it looks like the synonym configuration may be broadening the search. Your Complaint synonym includes terms such as rude, employee, yelled, unsatisfied, poor, and behavior. You are also using a Boost by Synonym Dictionary rule with weight 100.

I would first disable the synonym and boost rule temporarily and test the same query again. This will help confirm whether the irrelevant results are coming from query expansion or ranking.

Also, your Scripted REST API is returning 5 results. If you want to return only highly relevant results, I would check the response handling in the script as well, rather than adding an arbitrary relevance threshold.

This should help narrow down whether the issue is with the synonym, boost rule, or API result handling.

*************************************************************************************************************************

If this response helps, please mark it as Accept as Solution and Helpful.

Doing so helps others in the community and encourages me to keep contributing.

Regards

Vaishali Singh
Linkedin - https://www.linkedin.com/in/vaishali-singh-2273361bb