Content Creation for Relevancy in AI Search

[Heather Phipps]

AI Search is ServiceNow's flagship search offering, featuring sophisticated linguistics, query intent detection, question answering for surfacing precise answers to user queries, and unified results across siloed content. Out of the box, before any tuning, AI Search provides a ~10% relevancy lift over Zing text indexing and search.¹  While machine learning–based relevancy will automatically optimize AI Search results over time, the following techniques may be used to maximize relevancy. They are especially useful for bootstrapping, while user interactions are not yet available to learn from.

 

Relevancy Ranking in AI Search

AI Search relies on a variety of content features for relevancy ranking of search result documents. These include:

• Title match — how well the query matches the document’s title
• Content match — how well the query matches the document’s body content
• Keyword match — how well the query matches the document’s tags or meta
• Keyword popularity — the popularity of the document’s tags or meta
• Document freshness — how recently the document was created and/or updated
• Document popularity — how often the document has been viewed

The title, content, and tags features weight terms by inverse document frequency, meaning terms that carry more meaning such as “Webex” and “ESPP” will have higher weights than more common terms such as “policy”.

Please keep in mind that we are always improving our algorithm and model features, so the information above is subject to change in future releases! 

 

Content Creation Best Practices

With this background in mind, we recommend the following practices for maximizing relevancy and empowering self-service:

1. Create descriptive document titles unique to each article. Title matches carry more weight than content matches in the default relevancy ranking model.
2. Take advantage of synonym dictionaries for higher recall. Synonyms expand search queries to include additional terms equivalent in meaning.
3. Clean up HTML heading and paragraph tags for an optimal Q&A experience. Use the <h1> to <h6> tags for headings. Split longer paragraphs into shorter, more digestible chunks of information with corresponding <p> tags.
4. Feel free to continue using the meta field to annotate Knowledge articles with pertinent keywords that may not appear in the title or content. As machine learning–based relevancy collects and learns from user interactions, manual annotation will become less necessary.

¹(AI Search NDCG) - (Zing NDCG) as measured on hand-labeled golden sets.

[LenaLindel_f]

Thanks for this article - a thought for the future - you write "Looking ahead to San Diego and Tokyo, we’re exploring adding user context features, such as office location, title, and position in the organization tree" - can I suggest adding organizational belonging to that? It is far more plausible that me and a collegue in the same IT group are looking for the same article, catalog item etc .. food for the thought process

[Inactive_Us1921]

@Heather Phipps How do we actually use this script include once it is imported into our instance via the XML. Is this run as a one off via a background job? What would we be the recommended steps to use this Script include?

We are needing to migrate our meta to tags as we have started using AI Search.

Kind regards

Sean

[Gerard Dwan]

Hi Sean, 

Great question! Head to 'Script - Background' and run the following to execute the script: 

new migrate_kb_meta_to_tag().migrateKBMetaToTag();

This will be a one-time operation. After you've done this, if there is still a need to add/update tags, do that directly in the tags field. 

All the best,
Gerard

[Heather Phipps]

We adjusted the instructions to hopefully make them more clear. Thanks for the feedback.

[Sean50]

Hi Heather thanks for that I have successfully run the script to copy any meta data over to the tags field. In terms of Indexing for tags is this something that is setup by default or are there steps to follow in order for ai search to use the tags field in order to show results based on the tags in the esc search.

I have run some test knowledge articles but do not seem to be picking up results based on new tags I have created?

[Community Alums]

Is there a way (currently) to include knowledge uses in to the result calculations? For example, when a user has found an article, it deflected an incident and the user clicks this helped OR if a Service Desk analyst attaches an article to an incident.....

[Gerard Dwan]

Hi Sean, 

You may want to check that you're indexing tags. 

Best,
Gerard

[Heather Phipps]

This best practices article has been updated based on customer feedback and product changes slated for Tokyo. In Tokyo, we will start indexing meta by default for Catalog Item (in addition to Knowledge), treat meta as multi-valued, and add a corresponding model feature with sufficient default weight for meta to influence relevancy.

[Heather Phipps]

Not today, unfortunately, but it's on our roadmap.

[Rob Sestito]

Hey @Heather Phipps - 

Great article here - thank you!

I wanted to give your method a try (the background script) to see if I an better tune our AI searching.

Now, I am not sure if these messages I am about to show you means our organization is already doing something similar to what you have here, and it is being handled in an area I am not sure of (I have only been in this organization for one year - and I just recently migrated the HR portal to the new EC Pro with AI Search).

This is a small screenshot of what I really have, the list goes far down.

Are you able to speak to this at all to help me understand it more.

Not sure if this is combined, but once I noticed that meta data was not indexed, I created that myself per this idea post: 

AI Search: Allow 'meta' field to be included

Would the above screenshot messages be due to the fact that I already created indexing for the meta data field?

Thank you so much in advance and for your time!

Cheers,

-Rob

[StefanoZ]

Hi @Heather Phipps 

If we introduce right now what is described in this guide, once we upgrade to Tokyo release will we need to disable newly introduced Business Rules that keep meta synched with Tags?

[Neil P]

@Heather Phipps 

This is an excellent question that needs to be addressed.  If I go through all of this to fix it now, what happens when Tokyo comes?  Do we need to identify and revert these changes before the upgrade?  Will the upgrade assume these steps have been taken and act accordingly?  

I don't want to fix it, just for the actual fix to break it again or cause a non-standard configuration that I now have to support.

[Heather Phipps]

In Tokyo, we introduced a new ml relevancy model feature, "keywords", to replace the existing "tags" feature. It captures the union of the "tags" and "meta" fields. Having a unified feature limits performance impact and allows for seamless transition back and forth between "tags" and "meta" usage. So, if you already implemented the steps described here, you don't have to do anything upon upgrade to Tokyo. If you'd rather wait for Tokyo to get this behavior, the steps described here will no longer be necessary.

[Neil P]

@StefanoZ In case you didn't catch this reply.

Thank you @Heather Phipps, this is excellent news!

Neil P

[d_62]

Hello,

If I may, the business rule script is raising errors. Creating a tag visible by Everyone is not possible unless you have the role  global_tags_creator.

It is working in fine in Background script as it is run by an admin.

So my questions is the AI search feature will work if we create tg not visible by everyone ?

Thanks for your feedback 

[Timothy F1]

@Rob Sestito I received similar errors when I ran the script. It appears to be due to existing tags that have the same name as a tag that the script is trying to create.  "Training", "Windows", and "führungskräfte" are all tags I would expect to see created from normal operations, so they seem to line up with that assumption.

You could test if that's the case in your instance by looking to see what records have used those tags. 

[Neil P]

@d_62 I ran in to the same thing.  I had to grant that to all ITIL so new KB and Items get properly indexed.  I don't like everyone having that role, so I'm hoping that this gets fixed.

[d_62]

@Neil P  It could be avoided at indexed source level. The tags indexation could be configured to index also tags not shared by everyone. The code in the business rule should be updated to remove the visibility part to everyone.

Doc: Enable indexing of tags for an indexed source

@Heather Phipps am I right ? 🙂

Thanks

[Dan Brown2]

@Rob Sestito @Timothy F1 did you manage to find any solution to this when running the background script?  It does seem that if the tag is already created it fails, rather than just using it.

[Rob Sestito]

@Dan Brown2 - No, I moved on from using this. While I am also no longer with the company when I replied to this post, I just stuck with indexing what was needed from Indexed Sources and the search has been working great.

[Dan Brown2]

Thanks for the reply Rob. 

[Joakim St_ckel]

Hello @Heather Phipps and the rest of the ServiceNow community!
We're sitting with the AI Search active in our ESC Pro instance. We've receieved requirements to make the META tags functioning as search points in the portal. How do we make this happen? I've managed to find my way to the kb_knowledge.meta table but can't seem to figure out how to make sure the META tags are still working as search points.

BR, Jocke

[Erik Nelson]

We've recently upgraded to Tokyo but have been working a little on AI Search tuning since San Diego. Is the "new ml relevancy model feature, "keywords"" automatically enabled and is there a way we can validate that? Also, does this mean the warning message regarding "AI Search doesn't index tags you add to the Meta field..." can now be ignored? Thank you!

[Gerard Dwan]

Hi @Joakim St_ckel 

It was an automatic upgrade for customers. You should be able to add an arbitrary word to the meta field, wait a few minutes, then search for that word and see the article come up in results. To test this, you should use an uncommon term in your index. 

[Gerard Dwan]

Hi @Erik Nelson 

You should confirm that the indexed source for your Knowledge table doesn't explicitly disallow searching on the meta field (via a not_searchable property in the field mapping tab). 

Otherwise, yes, that message can be ignored. 

[Joakim St_ckel]

Hi @Gerard Dwan thanks for multiple assistance 😉

We're on the latest updates with the instance and the ESC patches. I tried for instance to add the META tag "monkey" to an article, and it wont display. Instead I get search results such as "Did you mean money" and hits on that instead.

With what did this automatic update come? So that I can look into this to make sure we got this upgrade in our instance as well.

BR, Jocke

[Gerard Dwan]

Hi @Joakim St_ckel, If you click on 'Search for monkey instead?' on that screen, you should see the result there. It is an interesting problem because monkey can be a misspelling of money and there is only the one time that monkey is mentioned. 

[mikeib182]

From the write-up above, are there any new updates that are specific to Utah release? Also, I read on the product documentation that AI results are enhanced based on an aggregation of user interactions. Are you able to comment on what user aggregations improve relevancy scoring on AI?

[Shamus Mulhall]

Hi @mikeib182 

Utah release updates: https://youtu.be/BZPKCcbL92w

Utah Prod Release notes: https://docs.servicenow.com/bundle/utah-release-notes/page/release-notes/now-platform-administration...

ML Relevancy in AI Search: https://docs.servicenow.com/bundle/utah-platform-administration/page/administer/ai-search/concept/ma...

[mikeib182]

@Shamus Mulhall thank you for providing those documents. Guess I'm still wondering as to what specific user interactions are added to the relevancy model. It stated: "These search signals include data on how search users interact with the search input field, auto-complete suggestions, facet and navigation tab filters, Genius Result answer cards, and search results."

In regards to knowledge articles, what happens if users rate articles or mark it helpful? Does this have any influence on the relevancy model? What if content is structured with a custom knowledge article templates with separately weighted fields? Maybe I'm not understanding how this works but would like some clarity.

[Mosheh]

Are meta/keywords or phrases in a knowledge article or catalog separated by a comma or underscore? - knowledge_article, knowledge article, 

[Randy TCTS]

@tangcov .  Check this out.

[C_S3]

Hello,
Can we change the sequence of Ranking in AI Search? We want meta to be at the top.
For eg : if we add 'manager' as meta in one article then when we search 'manager' in sp portal then our article should be at the top.

[C_S3]

Hello @Heather Phipps ,
Can we change the sequence of Ranking in AI Search? We want meta to be at the top.
For eg : if we add 'manager' as meta in one article then when we search 'manager' in sp portal then our article should be at the top.

[M_87]

We have set up the AI search on our instances and on top of that we extended the AI search with AI boost on Country filter (https://www.servicenow.com/community/ai-intelligence-articles/result-improvement-rules-for-global-co...)

Is it possible to have this prio over the current relevancy ranking?
Because results are still ordered by the current relevancy ranking and within each relevancy it orders based on country.

[Nisha15]

Hi @Gerard Dwan 
I have two queries can you please help:
1. Can we show the profile photo here on search itself?
2. Can we add email and designation below user name(currently shows user).

[Kass3m]

@Heather Phipps do the same rules apply to record producers and catalog items? because when we add a meta to that catalog item it says that this wont work if AI search is enabled.