Very wrong Genius Search Result - What can I do??

Chris D · ‎02-10-2022

Using the ootb Genius Search configuration and mostly ootb AI Search configuration as well (just Catalog Items + Knowledge). Since the Service Catalog is the primary use of our Portal, I have a slight boost setup for all catalog items and I'm also boosting one specific catalog item - though that's irrelevant to this case.

See below for my issue. The search term is "update distribution list" and understandably (with that cat item boost), our new KB article was appearing second in the results after the distribution list catalog item, so I created one more boost for that specific KB article when keywords contain "distribution list". Perfect - that bumps it up to the first result...

BUT WHY IS THE GENIUS RESULT SO BAFFLINGLY INCORRECT???

That genius result does not contain the words "distribution" or "list" anywhere within the catalog item - only the word "update" in the name.

And ultimately, what can I do about this besides disabling Genius Results altogether??

I'm thinking maybe NLU - again, the ootb (and read-only!!) Genius Result Model - is the culprit here since it seems to be matching the catalog item intent 100% for some odd reason (even though "update" isn't any of the utterances!), but I'm not sure how this helps me - it doesn't even tell me what the "distribution list" entity is mapping to. And if it did, why wouldn't it be mapping to the catalog item that has "distribution list" in the name (and short description and description)??

Chris D · ‎05-09-2022

Just to close this out, I did have this "resolved" by a ticket with ServiceNow.

The solution, albeit still not perfect, was basically just to upgrade from Quebec (to San Diego) since Genius Results have been improved notably since. From ServiceNow:

The model prediction quality is not as good as we hope in Quebec. That's why in Rome release we changed catalog genius result to be a search-based and won't depend on NLU anymore. Search-based catalog GR will not have this problem and has better accuracy.

...

If you upgrade to Rome or San Diego , you don't need to do anything. NLU based catalog Genius Results will be automatically changed to search-based.

Besides this setup, Rome also allows Result Improvement Rules to be used with Genius Results:

Result improvement rules now enable you to block unwanted Genius Results as well as search results. You can configure a result improvement rule to block only Genius Results and leave search results unaffected.

And they added Helpful/Not Helpful feedback for users in Genius Results as well to Rome:

Provide feedback on Genius Results

Submit feedback for each Genius Result answer card, specifying whether it's relevant to your search or not. AI Search records feedback responses as search signals that you can use to assess the relevancy of your Genius Result configurations.

View solution in original post

Mary Beth Hutch · ‎04-27-2022

This is also occurring in my instance; what I found was that in the "All" navigation tab, some of my items are not being returned at all - causing the result to be blatantly incorrect. If I switch to the "Service Catalog" tab, the result will be returned correctly.

I am working with ServiceNow at this time to address; if anyone has seen this and has an idea, please share!

UPDATE: I figured out my problem; I realized that my search source needed to be updated; so pay extra attention to Contains and Does Not Contain searches... it mattered!

Chris D · ‎05-09-2022

I actually had a separate issue myself actually with some results not being returned in the "All" tab and that was "fixed" from a ServiceNow ticket where they revealed that we can't have multiple search sources from the same source - i.e. one filtered "primary" KB source and a secondary "All Knowledge" source 😕

That fix was just eliminating the filtered source and using a single KB search source. As a workaround, I just used a result improvement rule to boost articles from our "primary" KB.