Unexpected behavior after upgrading to Australia

Go to solution
Mattie Gillenw
Tera Contributor

3 weeks ago

I installed ServiceNow Otto for IT Service Management (ITSM) on our Dev environment, running Zurich at the time. After playing around with the configurations, I built a custom AI Agent to be used with the LLM Virtual Assistant. It worked as intended. Whenever a user asked a question, the assistant would run the AI Agent which searches our KB and Catalog items. Then it would return those results in an easy to digest format. If it was unable to find any information on the subject, it would ask a follow up question and then offer to open and Incident or connect the user to a live agent. The process was smooth and user friendly.

 

Upon upgrading to Australia, the assistant no longer activates the AI Agent as its first response. Now it initiates what appears to be a default QnA protocol. The results from this are not as robust or formatted in a way that suits our needs. Then it will ether activate the custom AI Agent or tell the user they can use the AI Agent for a better result. 

 

I have been searching the net to try and find a way to prevent this QnA from activating and revert back to the AI Agent being the default resource for the assistant. I have tried removing all setup topics and creating a custom setup topic, as instructed from my searching. Creating the custom setup topic has given me the best results, but it will still pop up with that initial QnA response before starting the AI Agent about 50% of the time. 

Has anyone found a way to reliably get the assistant to start with a custom AI Agent?

0 Helpfuls
  • 396 Views
1 ACCEPTED SOLUTION

Go to solution
RajveerSingh
Kilo Guru

3 weeks ago

From what you have described, I believe this is more related to the AI Agent routing/discovery behavior in the Australia release rather than an issue with your custom AI Agent itself.

Since the same configuration worked as expected in Zurich, I would recommend checking the following areas after the upgrade:

1. Verify the application/plugin versions

Make sure the relevant Now Assist, AI Agent, Generative AI Controller, and Virtual Agent components are updated to the recommended versions for Australia. An upgrade can introduce changes to the way AI Agents and Q&A capabilities are evaluated.

2. Check the Assistant configuration

Go to the Virtual Agent/Assistant configuration and verify that your custom AI Agent is explicitly enabled for the Chat Assistant. Having the agent active in AI Agent Studio does not necessarily mean that it will always be selected as the first response by the assistant.

3. Review the AI Agent description and instructions

The agent's name, description, and instructions are important because they help the LLM determine when the agent should be used.

I would make the purpose of the agent very explicit, for example:

> "This is the primary IT self-service agent. For employee IT-related questions, first search the organization's Knowledge Base and Service Catalog. If relevant information cannot be found, ask a clarifying question and then offer to create an Incident or connect the user with a live agent."

This can improve the agent's discoverability and routing.

4. Review other active AI capabilities

After the upgrade, check AI Admin Center/Asset Inventory and look for any OOB Q&A, Search, Skills, or other capabilities that could handle the same type of questions as your custom agent.

If multiple capabilities are eligible for the same request, the assistant may choose Q&A first and then invoke your custom AI Agent afterward.

5. Check the execution/interaction details

I would compare two cases:

* A case where the custom AI Agent is selected immediately.
* A case where Q&A is selected first.

The execution details/trace should help identify why the assistant selected one capability over the other. This is probably the most useful way to determine whether the issue is related to routing, discovery, or configuration.

I would also avoid relying on deleting Setup Topics as the primary solution. That may change the behavior, but it does not necessarily guarantee that the custom AI Agent will always be selected first.

For your use case, I would ideally configure the custom AI Agent as the primary entry point for IT self-service, with the agent itself handling Knowledge Base and Catalog searches, followed by clarification, Incident creation, or live-agent handoff when necessary.


In short, I would focus on Assistant configuration + AI Agent discoverability + competing AI capabilities + execution traces, rather than continuing to remove Setup Topics.


Please Accept the solution if it assisted you with your question & Mark this response as Helpful.

Regards
Rajveer Singh Rathore
Servicenow Developer
Linkedin - https://www.linkedin.com/in/rajveer-singh-rathore-209b9683/

View solution in original post

0 Helpfuls
1 REPLY 1

Go to solution
RajveerSingh
Kilo Guru

3 weeks ago

From what you have described, I believe this is more related to the AI Agent routing/discovery behavior in the Australia release rather than an issue with your custom AI Agent itself.

Since the same configuration worked as expected in Zurich, I would recommend checking the following areas after the upgrade:

1. Verify the application/plugin versions

Make sure the relevant Now Assist, AI Agent, Generative AI Controller, and Virtual Agent components are updated to the recommended versions for Australia. An upgrade can introduce changes to the way AI Agents and Q&A capabilities are evaluated.

2. Check the Assistant configuration

Go to the Virtual Agent/Assistant configuration and verify that your custom AI Agent is explicitly enabled for the Chat Assistant. Having the agent active in AI Agent Studio does not necessarily mean that it will always be selected as the first response by the assistant.

3. Review the AI Agent description and instructions

The agent's name, description, and instructions are important because they help the LLM determine when the agent should be used.

I would make the purpose of the agent very explicit, for example:

> "This is the primary IT self-service agent. For employee IT-related questions, first search the organization's Knowledge Base and Service Catalog. If relevant information cannot be found, ask a clarifying question and then offer to create an Incident or connect the user with a live agent."

This can improve the agent's discoverability and routing.

4. Review other active AI capabilities

After the upgrade, check AI Admin Center/Asset Inventory and look for any OOB Q&A, Search, Skills, or other capabilities that could handle the same type of questions as your custom agent.

If multiple capabilities are eligible for the same request, the assistant may choose Q&A first and then invoke your custom AI Agent afterward.

5. Check the execution/interaction details

I would compare two cases:

* A case where the custom AI Agent is selected immediately.
* A case where Q&A is selected first.

The execution details/trace should help identify why the assistant selected one capability over the other. This is probably the most useful way to determine whether the issue is related to routing, discovery, or configuration.

I would also avoid relying on deleting Setup Topics as the primary solution. That may change the behavior, but it does not necessarily guarantee that the custom AI Agent will always be selected first.

For your use case, I would ideally configure the custom AI Agent as the primary entry point for IT self-service, with the agent itself handling Knowledge Base and Catalog searches, followed by clarification, Incident creation, or live-agent handoff when necessary.


In short, I would focus on Assistant configuration + AI Agent discoverability + competing AI capabilities + execution traces, rather than continuing to remove Setup Topics.


Please Accept the solution if it assisted you with your question & Mark this response as Helpful.

Regards
Rajveer Singh Rathore
Servicenow Developer
Linkedin - https://www.linkedin.com/in/rajveer-singh-rathore-209b9683/

0 Helpfuls