Deploy, Monitor and Improve AI Search
Once AI Search is configured for your use cases, it is time to deploy the AI Search configuration in prod. After go-live, it is recommended to use analytics to monitor the solution. This will help gain insights into users’ behavior and needs to drive improvements.
Deploy
Deploying the AI Search configurations from dev to sub-prod and prod environments follows the same process as deploying other applications. The first step is to activate all the plugins on each instance. The second step is to migrate the configuration via Update Sets or Source Control.
Once the configuration is migrated:
- Re-index the AI Search Indexed Sources on the destination instance
- Publish the Search Profiles
- Sync the Vocabulary Sources,
- Then train and publish the NLU models on the destination instance
Use analytics to monitor user’s behavior
Monitoring can be done in prod with the help of the User Experience dashboard, delivered with the Service Portal Analytics plugin [com.glide.service-portal.analytics].
The portal where AI Search is used might need to be enabled to collect Analytics; follow these steps.
Navigate to User Experience Analytics > Dashboard. Select the application to monitor (for example, Employee Center). Then, select Search Analytics.
This dashboard helps review search trends to understand the needs of search application end-users better and provide insights on how to improve the search experience.
We will focus our attention on two metrics: “Queries with no clicks “ defined as “the most frequently submitted search queries in the selected application and date range for which users didn't select any search result, by count” and “Queries with no results “ defined as “the most frequently submitted search queries in the selected application and date range that produced no results, by count.” By default, only the first 5 records are displayed, click on the “View All” button to display all the queries.
For more details about the content of the dashboard, refer to the documentation.
Note: There is another set of dashboards delivered as part of the Advanced AI Search Management Tools plugin, available through AI Search > AI Search Analytics. These dashboards provide an overview of the search configuration and number of queries, but the information is not granular enough to get insights into users’ behavior.
The Search Profile Analytics dashboard provides metrics regarding a Search Profile for a given time frame. It shows the number of searchable records (called documents) and their breakdown by Search Sources. It also shows the number of queries run against this profile, broken down by Search Application and over time.
The Search Index Analytics dashboard provides metrics about the total number of indexed records, their breakdown by Indexed Sources and Search Profiles and the trend over time. It also shows the overview of the configuration. The second tab (AI Search Query) provides metrics regarding queries by Search Profiles, languages and aggregated over time.
Apply Continuous Improvement
Let’s review what to improve based on the insights from the User Experience dashboard.
When a query didn’t return results or didn’t lead to a click, it could be for one of these reasons:
The query is irrelevant or out of scope – a user is looking for content outside the scope of your application (e.g., looking for an area of your business not covered by the portal); the appropriate content neither exists nor is it exposed through search. Additionally, the query could have been clearly misspelled.
What to do about it?
This is expected (True Negative). If there is a trend, use the data to plan for expanding your coverage.
Content is missing – a user is looking for content that should be there but doesn’t exist today.
What to do about it?
If many users are looking for content that doesn’t exist, it could be a sign that additional content should be created to address that gap.
Content exists but is filtered out – a user is looking for existing content but is not exposed through search.
What to do about it?
Start by checking the filters on the Search Sources to make sure the right content is exposed through search. Also, check that the permissions on specific records are still relevant (e.g., users allowed to view a Knowledge article). Review the synonyms and stop words to ensure terms are appropriately listed.
Content is not considered relevant for this query – a user is looking for existing content, exposed through search but is not returned for a specific query.
What to do about it?
Start by reviewing the content itself (e.g., is the description of a catalog item still relevant), and review these best practices.
Use Result Improvement Rules to promote and boost appropriate search results based on search terms or user context.
Recommendations for deploying improvements
As discussed, opportunities for improvement can be found by monitoring the dashboards in a prod environment. Changes can then be planned, built in a sub-prod environment and deployed as part of your organization's usual release process.
What's next?
You have now completed the configuration and deployment of AI Search, continue to monitor to find ways to improve!
Go back to the Quick Start Guide.
References
 |
Documentation |
 |
FAQ |
Don't forget to subscribe to our AI & Intelligence forum to get the latest updates!
