Generative AI Controller FAQ

[Victor Chen]

(Washington DC release)

1. What is the Generative AI Controller? 

The Generative AI Controller is our intelligence connection layer to the Large Language Models (LLM) that produce AI-generated content. It serves as the foundation for our generative AI products but can also be leveraged to create your own custom generative AI applications and workflows within Flow Designer, Virtual Agent Designer, and scripts. 

You can find an example of building and deploying a custom workflow here.

2. Where can the Generative AI Controller be used in the Now Platform? 

You can leverage the Generative AI Controller to integrate generative AI functionality within Flow Designer, Virtual Agent Designer, and general platform scripting.  

Example usage within Virtual Agent Designer:

3. What out-of-box use cases are provided by the Generative AI controller app? 

The use cases available are below: 

Action	Description
Generate Content	Generate content with generative AI to create bodies of text quickly, such as generating proposed responses agents can use to respond to incoming emails.
QnA	Help users find the answers to their questions, freeing up time for agents
Summarize	Summarize large bodies of text or complicated exchanges to make the transfer of information easier, such as during handoffs between Virtual and live agents
Generic Prompt	Use Generative AI to generate ideas and brainstorm on any topic, such as ideas for increasing engagement. E.g. you can ask the LLM to create a marketing plan.
Sentiment Analysis	Calculate the sentiment of an input utterance as Positive/Neutral/Negative.

4. Which Generative AI providers are currently supported? ​​​​​​​ 

        OpenAI, Azure OpenAI, Aleph Alpha, IBM WatsonX, and Google Cloud (Vertex AI and Makersuite). As of the Washington release, you can also connect with any external LLM using the generic LLM connector.

5. What are the requirements to get started with the Generative AI Controller?  

To begin using the Generative AI Controller, you need the following:  

• An existing Pro Plus/Enterprise Plus entitlement
• License and API key from the external LLM you wish to connect to
  
• Your instance must at least be on Vancouver P2+

Store app:   

• Generative AI Controller  

Note: You do not need to install the controller store application yourself - it is bundled with the Now Assist for [x] plugins.

6. Does the Generative AI Controller use Integration Hub and consumes transactions?
The Controller uses embedded Integration Hub spokes to connect to the third-party LLM service providers, but uses does not incur IHub transaction costs. It will consume "Assists" as part of the Pro Plus/Enterprise Plus SKU. 

7. Does my ServiceNow data leave my instance when using 3rd party Generative AI providers? 

        ​​​​​​​Yes. Please be aware that data and queries you make from the app are sent to the party that hosts the external LLM you are leveraging. Note their data privacy policies and decide what usage policy best fits your organization. 

You can learn more about how Now Assist handles data handling and security here.

8. I have sensitive data in my instance that I don't want to leave my instance - how can I do this?

You can employ the use of the Sensitive Data Handler plugin. It is not active by default and must be installed and configured by the admin to help detect and mask data considered sensitive for your business. To configure, you must define the regex patterns that match what you consider to be sensitive information.

9. Is the generative AI output moderated for unsafe or harmful content? 

        ​​​​​​​We released Now Assist Guardian in Q4 2024 to provide content moderation and adversarial attack detection guardrails for Now Assist skills. For custom skills and providers, this depends on how you configure your account/instance of the external LLM. Each provider will have their own content moderation abilities that you can review and determine if that are fit for purpose.

10. Is there a possibility that Generative AI output is inaccurate (e.g. hallucination)?  

        ​​​​​​​While we have minimized the possibility of hallucinations through selective use cases and prompt engineering, there is always a risk of inaccurate information in generated content. Thus, it is always recommended to employ human-in-the-loop review for such content.  

11. Is it possible to modify/create my own prompt, temperature, or number of tokens? 

        ​​​​​​​Currently no. ServiceNow provides a read-only initial prompt, and in your workflows you can expand upon it. Default temperature = 0. Default max_tokens = 500 is the amount reserved for the response.  

12. What models are currently supported out-of-box for OpenAI? 

        ​​​​​​​GPT-3, GPT 3.5 turbo, and GPT-4.  

13. What is the external cost per transaction calling the API?​​​​​​​ 

        It depends on the provider and the length of the output. We recommend reaching out to the providers to understand their pricing models.

14. When using Azure OpenAI, I’m running into an error: "The API deployment for this resource does not exist. If you create the deployment within the last 5 minutes, please wait a moment and try again."  

        Ensure that your deployment name is the same as the model name in Azure admin portal. 

15. Is it possible to maintain context between multiple queries?  

            Not today.

16. Can I use the Generative AI Controller to connect to Now LLM for my custom use cases?

Currently no. You can only connect to external LLM providers with the Generative AI Controller. 

[Dan Y25]

Hello and thank you for the good content.

For item 12, "What is the external cost per transaction calling the API?​​​​​​​", does the answer refer to "Custom use cases built using the Gen AI Controller that are not part of the Now Assist experience"?

The reason I ask is that I saw what seemed like a conflicting statement in a different GenAI FAQ document.

Thank you, Dan

[Jonathan Demeu1]

Hi, thanks for the content.

We have just recently upgraded our ITSM Standard to ITSM Pro. Does that mean that we will not be able to use any of the new Now Assist content?

We are also paying for IntegrationHub Professional.  I would be saddened to hear we need yet another premium to use these features.

Kind regards,

[Lener Pacania1]

Johnathan, you will need to purchase a Pro+ license to access any of the new Generative AI capabilities such as Now Assist.  The core requirement is Vancouver Patch 2 and a PRO SKU license(i.e. ITSM PRO).  Unlike other products, ServiceNow will continue to add new GenAI features to the Pro+ SKU via store updates.  For example we released on 9/29 Now Assist for ITSM which provides incident, live agent chat summarization, KB summarization,  and resolution generation.  In November we will add additional new GenAI features.    You can discuss this further with your ServiceNow account team.  -Lener

[Jerry kk Wong]

Is the Generative AI Controller available to use on a PDI or development instance to try out these features?

[Lener Pacania1]

Jerry, unfortunately not on PDIs or even sub-production instances.  The only way to test out ServiceNow's GenAI is to work with the ServiceNow account team and request an evaluation.  The reason is the GenAI prompts that hit the Now LLM consume GPU resources and it's expensive, so we can't open it up as we have in the past with other platform capabilities.

[Victor Chen]

@Dan - yes it does refer to just custom built use cases, not Now Assist. 

[Tomas14]

Will there be difference in "Assists" consumption depending whether external LLM is used or Now LLM (in the future)? I assume use of Now LLMs will consume GPU processing power installed in ServiceNow datacenters whilst with external LLM it's the 3rd party who provides GPU processing power.   

[Raffaele Bologn]

Hi Team, are our partners currently allowed to try general-purpose Gen AI (Open AI/MSFT Azure Open AI) from their demo instances? Please confirm, thanks again Raf

[Tim D Scott]

Hi @Victor Chen we are a ServiceNow tech partner. Is the Gen AI controller available on our vendor instances?

[Community Alums]

Hi All, to generate QnA summarizations on Portal and VA how can we use azure open ai? We use Q&A - Now Assist for Search to retrieve summarized responses on portal search and va search which uses the NOW LLM. How do i make it return summarizations using gpt 3 or 3.5 model?

Thanks,

Sandesh

[GChanner]

Hi Victor,

I am in the healthcare space in a recent webinar SN stated it will not roll out GenAI to healthcare customers due to HIPAA. Once SN knows GenAI has met HIPAA compliancy then it will be available for healthcare customers. Is this correct and do you know of a timeline?

[GChanner]

Will GenAI/NOW Assist be able to tell a group manager how many incidents are still active/opened for a given time period

[Brian Bakker]

@GChanner 

We already have Natural Language Query to query the data in your instance by entering plain language requests into the user interface. However, we will continue to add more GenAI capabilities/actions into Now Assist to make it more feature rich, so watch this space.

[GChanner]

Thank you Brian. Your response is very helpful. We have been receiving a lot of inquiry from our user base about GenAI and as such we are accessing its value/ROI to the organization.

[JADrisko]

@Victor Chen I have a question on something that wasn't mentioned in your great FAQ above:

What’s ServiceNow’s position on our clients building their own API integrations to OpenAI or Microsoft OpenAI (or their own LLMs) without using the Generative AI controller?

Technically these systems are just REST APIs that can be connected to.  They’d lose all the pre-built actions in Flows and have to do everything themselves to maintain it, but would this still require the Now Assist licensing with the GenAI controller?

[Robbie]

Hi @Victor Chen,

Thanks for great article - very helpful.

I wonder if you can answer a question I have around obtaining Sentiment Analysis specifically for HR cases. Whilst I see it available 'Out of the box' for CSM, and ITSM, as well as KB and being able to configure it for Flow Designer... is there anything available for HR Cases please? I've spent hours going through articles and docs and related sites but I can't find anything to confirm Sentiment Analysis availability for HR.

Thanks, Robbie

[Markus B]

How can Amazon Bedrock (https://aws.amazon.com/bedrock/) be integrated and leveraged as another generative AI capability?

Thanks, Markus

[Stefan3000]

Hi @Victor Chen ,

great article thank you very much.

Currently, much of AI personalization can only be achieved through creating custom REST calls.

(Prompt Engineering, Parameter Tuning....)

As @JADrisko already asked: Could you please clarify whether it's permissible to create custom REST calls via IntegrationHub?

Thank you very much!

Greetings from Germany 🙂 

[Flavia Margaret]

hi @Victor Chen, may I know if we need to have a separate resource of OpenAI or Azure OpenAI to get the Now Assist capabilities to work? I was trying to test the flow you have in this youtube video (Link), but since I don't have the connection configured as I don't have Azure OpenAI resource, the testing failed with error message "Unable to load connection with alias ID: xxx".

I thought we could just use the Now LLM to get the Now Assist to work, but apparently no? Your clarification is truly appreciated, thank you!

[Victor Chen]

@Flavia Margaret - You can't use OpenAI with Now Assist capabilities, only the four generic ones.

[E Chow]

Good FAQ. Please clarify #1 and #16. Can I connect the GenAI Controller to a custom application I configured on the Platform. eg, I created an app in year 2000 with custom tables and workflows and want Now Assist to summarise specific records.

[nikkii]

Hi guys

Thanks for this article. 

I read in other articles about the feature "Bring your Own LLM" with the Generative AI Controller.

Is this already available? Can't find any Information on servicenow Doc, nowlearning or community..

Thanks to all.

[Brian Bakker]

@nikkii 

Please review our documentation Configure a generic large language model (LLM) connector

[nikkii]

Hi @Brian Bakker 

Thanks for adding this link.

That explains, why i dont find any infos for the term "Bring your own LLM" 

Thought that is a feature / product, not just wording.

So the funktion custom LLm's is already released isnt it? 

Or is there coming more?

Thanks

[Brian Bakker]

@nikkii 

"Bring your own LLM" might be a marketing term 😊

ServiceNow will be LLM agnostic, and we will support any LLM of your choosing is my understanding of our offering, but it will require an LLM connector. We recently announced the ability to connect Amazon Bedrock models to ServiceNow, so for sure we will be adding plenty of support for other LLMs that you can invoke through the Now Assist Skill Kit, where you can create your own prompts and incorporate them into your AI Agents.

Regards,

Brian