Very wrong Genius Search Result - What can I do??

[Chris D]

Using the ootb Genius Search configuration and mostly ootb AI Search configuration as well (just Catalog Items + Knowledge). Since the Service Catalog is the primary use of our Portal, I have a slight boost setup for all catalog items and I'm also boosting one specific catalog item - though that's irrelevant to this case.

See below for my issue. The search term is "update distribution list" and understandably (with that cat item boost), our new KB article was appearing second in the results after the distribution list catalog item, so I created one more boost for that specific KB article when keywords contain "distribution list". Perfect - that bumps it up to the first result...

BUT WHY IS THE GENIUS RESULT SO BAFFLINGLY INCORRECT???

That genius result does not contain the words "distribution" or "list" anywhere within the catalog item - only the word "update" in the name.

And ultimately, what can I do about this besides disabling Genius Results altogether??

I'm thinking maybe NLU - again, the ootb (and read-only!!) Genius Result Model - is the culprit here since it seems to be matching the catalog item intent 100% for some odd reason (even though "update" isn't any of the utterances!), but I'm not sure how this helps me - it doesn't even tell me what the "distribution list" entity is mapping to. And if it did, why wouldn't it be mapping to the catalog item that has "distribution list" in the name (and short description and description)??

[Greg K]

I am very interested if anyone knows the answer

[Lener Pacania1]

Hey Chris,

I verified the same issue on my Rome instance.  The behavior should be like the below, when I enter something like "I want a cheeseburger" the ootb AIS NLU model returns confidence below threshold and the Genius Result(GR) will not pop.  I'm not sure why "update distribution list" is returning 100% catalog find for the predicted intent.  I do know we disable GR out of the box as AIS PM is working on increasing the precision of the GR NLU in San Diego.

I will send a note to the AIS PM team and see if they can jump on this thread. Do you have an ticket open?

[Chris D]

No ticket yet - figured I'd check the Community first in case I was missing something obvious.

[Chris D · Accepted solution]

Just to close this out, I did have this "resolved" by a ticket with ServiceNow.

The solution, albeit still not perfect, was basically just to upgrade from Quebec (to San Diego) since Genius Results have been improved notably since. From ServiceNow:

The model prediction quality is not as good as we hope in Quebec. That's why in Rome release we changed catalog genius result to be a search-based and won't depend on NLU anymore. Search-based catalog GR will not have this problem and has better accuracy.

...

If you upgrade to Rome or San Diego , you don't need to do anything. NLU based catalog Genius Results will be automatically changed to search-based.

Besides this setup, Rome also allows Result Improvement Rules to be used with Genius Results:

Result improvement rules now enable you to block unwanted Genius Results as well as search results. You can configure a result improvement rule to block only Genius Results and leave search results unaffected.

And they added Helpful/Not Helpful feedback for users in Genius Results as well to Rome:

Provide feedback on Genius Results

Submit feedback for each Genius Result answer card, specifying whether it's relevant to your search or not. AI Search records feedback responses as search signals that you can use to assess the relevancy of your Genius Result configurations.