Going Live with AI Search? Follow these steps

[Gerard Dwan]

As many customers prepare for going-live with AI Search, there are a few best practices steps to ensure a smooth go-live and first few weeks.

AI Search Go-Live Checklist

• Enable AI Search in production in advance
  • Logged in users can review the AI Search provisioning process here
• Identify internal content and the filters to apply in order to define Indexed and Search Sources
  • Only index documents that will be searchable in at least one Search Source (active=true, published=true etc.)
• Install the Advanced AI Search Management Tools store app
• Define metrics that are important to deliver upon for your use of AI Search
• Perform benchmarking activities in a lower environment to create a baseline for these metrics
• Test the experience by impersonating users in the portal that is going live with AI Search
  • Be sure to click on a variety of search results (at least one per ServiceNow table)
• Consider setting up Result Improvement Rules to boost or promote content and guide users to the most appropriate results. For example, if the portal includes People Search, people results with vip=true can be boosted or, when user’s search for “CEO” or his/her name, the CEO’s result can be promoted to ensure it is the top result.

There are processes that will help make AI Search perform better over time. Namely, automatic relevancy training which should improve results monthly. Review the Search Analytics Dashboard’s “Average Click Position” report. If Machine Learning Relevancy is doing its job, the average click position is expected to trend towards position 1 as it learns about users’ preferences and tunes its models.

There are steps that you can take, however, to improve the experience beyond that. They are listed here:

• Review the Search Analytics Dashboard's "Searches with 0 results" report for any synonyms which should be added to the Synonyms dictionary to improve matching. For example, if users are searching for "Ping ID" but the actual product name is "PingID", a synonym entry associating these variations would help users find what they're looking for.
• Review the Search Analytics Dashboard’s “Searches with 0 clicks” report to identify gaps in content. These are searches that users are executing but not clicking on a single search result. This could identify a subject area which is not adequately covered by your Knowledge Base or Service Catalog. This report may also suggest the need for additional synonym or stop words entries.
• Add entries to the Blocked Terms for the Default Spell Check Dictionaries. For example, if a term or name is being detected as a typo when it is not, adding it to the Blocked Terms list will prevent it from being auto-corrected.
• Consider adding the company name to the Stop Words dictionary, especially if users get less relevant results when they include it in their search queries. Test the impact on a range of queries.
• Consider setting up Result Improvement Rules to de-boost or block outdated content. Content can be pushed lower in the results or blocked entirely either by individual document or in bulk.