Wildly Inaccurate Results Using NowAssist Searches

[briannuernb]

We recently enabled Now Assist for our fulfillers and, overall, many of the core capabilities are working as advertised. Features such as record summarization, content generation, and email generation have generally been well received by our teams.

Where we're running into challenges is with the Now Assist Panel's search and retrieval capabilities, particularly when users attempt to query ServiceNow data using natural language.

A few examples:

• When asking Now Assist to return remediation tasks associated with a specific Configuration Item (CI), it consistently returns records tied to different CIs than the one requested.
• When asking for incidents I've opened in the last 12 months, it may return problem records instead of incidents, or it appears to use my name in an unrelated field as part of the search criteria(see attached).
• In general, the results often seem disconnected from the intent of the prompt and can be difficult to trust without manually validating the returned records.

We're trying to better understand whether this is expected behavior, a configuration issue, or simply an area where prompt engineering is required.

For those who have successfully deployed Now Assist:

• Have you found ways to improve the relevance and accuracy of record retrieval from the Now Assist Panel?
• Are there specific configurations, indexing considerations, data source settings, AI Search tuning, or or other configuration/techniques that made a noticeable difference?
• Have you identified prompt patterns that consistently produce better results?
• Are there any known limitations with querying transactional data (Incidents, Problems, Tasks, Vulnerabilities, etc.) that we should be aware of?

We're currently evaluating adoption across a broader user base, and the search experience is one of the biggest areas preventing wider confidence in the tool.

I'd appreciate any lessons learned, best practices, or troubleshooting recommendations from others who have been down this path.

[Tricia Cornish]

@briannuernb 
It could be a few issues. The NAP is a VA assistant and works like a VA conversation in an agentic wrapper so I try to troubleshoot from sys_cs_conversation to see what topics are being retrieved in the sys_cs_conversation_task. 

In the latest releases, VA prioritizes AI Agents over the other content defined in the assistant - that's the agentic wrapper. The VA conversation is an Agentic workflow (calling default VA workflow), so you could also look at sn_aia_execution_plan to see how your query executed. 

I know I had to add very specific description identifiers that agents are only called when specific phrases or words are used. Some of the out-of-the box agents and skills have closely related descriptions. It might be helpful to identify AI agent usage v standard Q&A usage and define your roles for all the content available in the assistant accordingly. As an example, let's say your Tier 2 analysts will use the AI Agents + query capability (and trained to be a bit more specific in their query asks) and your platform users and Tier 1 analysts (who might use looser language and who are using the skills in the workspace) will not "discover" the topics and already used skills. For them, Panel might be more Q&A where you're surfacing content from the KB, catalog, graph and the triage and troubleshooting agent.  

What might be helpful is to look at the search log (sys_search_event) and add columns for the search profile, search application and user context - these will tell you the profile that is actually being called in your query. With enhanced chat and enhanced Q&A, you can start a search in Global Experience or a Workspace Search, then ask a followup (which opens the panel).  This might also help you to direct users to use, say, workspace search for table-based queries and Q&A search, and the Panel is restricted to agentic launching. (just an idea)

Another item to look at are the graphs activated in your Panel Assistant - this might be the CI query issue. If you haven't explored the knowledge graph capabilities, this allows users to query data from tables (including task records) without having to create an agent, VA topic or specific skill to do data retrieval. A default Knowledge Graph for the panel is set in the sys_properties (sn_vad_genai.nap.kg.schema.name) . (https://youtu.be/PrJBgOPyFfk?si=tIqnGWpqJ-qBoBZf )
There are a few other Academy sessions on graph, and for your CI query, you may want to look at any enhancements for search that includes the cmdb tables  - that would be enterprise graph.

It seems like you have closely related skills and if you have defined synonyms or queriable fields in your nodes, if could searching for the right content, in the wrong column. I would start with the search profile being called (sys_search_event) and the assistant content that is being sorted through to answer the query. AI Agent executions might help, in addition to the test preview window.
Hope that helps - update your post with any progress hits or misses 🙂
Regards ,
Tricia