- Post History
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
on
10-29-2024
03:22 PM
- edited on
11-01-2024
10:01 AM
by
Eliza
Overview
Now Assist Skill Kit (NASK) allows you to create new generative AI skills using the generic Now LLM or your own LLM. NASK has been designed to easily manage and integrate new skills into the ServiceNow platform via a UI Action and soon into the Now Assist Panel. With NASK you define the input data, leverage tools to process the data, activate the new skill, and deploy it to the platform (see below).
Using NASK to Automatically Categorize KCS KB Articles
A key challenge in building and maintaining an effective Knowledge Base for diverse self-service use cases is achieving accurate content categorization and tagging. Properly tagging articles based on their content and aligning them with a broader taxonomy enables users to quickly find relevant information, improving the self-service experience and lightening the load on support teams. Moreover, well-organized content prepares your organization for future AI-driven use cases, such as use cases relying on Retrieval Augmented Generation and advanced search patterns, unlocking greater value from your knowledge assets.
Given this challenge, Large Language Models (LLMs) are highly effective at interpreting unstructured text and extracting meaning. We can harness this capability by using NASK to process unstructured Knowledge Centered Service (KCS) KB article data and allowing the Now LLM to intelligently categorize the content by selecting from predefined categories within our Knowledge Base…effectively automating the categorization of KCS KB articles at massive scale (see high level flow below).
Below is what the finished product looks like:
1. The UI action calls the custom NASK skill.
2. Now LLM predicts category given KCS KB article text.
3. If Now LLM output matches existing category, update the category field. If no category match can be made, do not update category field.
You will need the following to implement this use case in your instance:
- Purchase any Now Assist SKU or request a Now Assist evaluation instance from your ServiceNow Account team.
- Upgrade the instance to the latest version of Xanadu and install any of the Now Assist plugins.
- Configure Now Assist Skill Kit by following the steps in our documentation.
- To use Now Assist Skill Kit you will need to grant yourself sn_skill_builder.admin role.
- Since we will be analyzing surveys the kb_knowledge and kb_category tables will need to be populated with KCS articles and corresponding article text.
Lab Flow:
There are nine phases to build a custom skill with NASK that automatically categorizes KCS KB articles.
- Phase 1 – Review knowledge base and gather which categories we want the Now LLM to use to categorize KCS KB article text.
- Phase 2 – Identify and review a specific KCS KB article that we can use as a test record for the custom skill.
- Phase 3 – Using data from Phase 1, build a Subflow that automatically gathers the categories we want the Now LLM to use.
- Phase 4 – Build the custom skill using NASK.
- Phase 5 – Testing the custom skill.
- Phase 6 – Deploying the custom skill
- Phase 7 – Activating the custom skill
- Phase 8 – See custom skill in action, on a single KCS KB article
- Phase 9 – See custom skill in action, on a list of KCS KB article
Phase 1: Review knowledge base and gather categories for use in NASK
First, we need to identify, review, and collect information about 1) the knowledge base and 2) the categories contained in this knowledge base for NASK use. The purpose is to capture a finite list of categories that the Now LLM can choose from given the text of a KCS knowledge base article.
1. Type kb_category.list into the filter navigator.
2. Identify and review which knowledge base you intend to use. For this demonstration I will use: Knowledge Base: IT in the Parent ID column.
3. Identify which knowledge base you intend to use in the Parent ID column. For this demonstration I will use: Knowledge Base: IT.
4. Next, right click on the Knowledge Base IT field of a record and click Show Matching.
5. Make note of the Parent ID (which should be a combination of characters and numbers) after clicking Show Matching. Using the lab example, the Parent ID for Knowledge Base: IT is a7e8a78bff0221009b20ffffffffff17. Store this ID for use later.
6. We also need to add the categories for another knowledge base. To do this, I go to the list of kb_category and repeat the above steps. For this lab, click Show Matching once more and note the Parent ID for Knowledge Category: Operating Systems: 9d4737cbff0221009b20ffffffffff33. Store this ID for use later.
Phase 2: Identify and review KCS knowledge article for use in NASK
1. Type kb_knowledge.list into the filter navigator.
2. Filter the list of knowledge articles so that class is KCS Article.
3. Identify KCS knowledge article(s) you intend to use as test records for NASK. The ideal candidate record should have Knowledge base as a required field. Additionally, Category should be (empty) or incorrect, and detailed text descriptions in the following fields:
Short description, Issue, Environment, Cause, and Resolution.
If you don’t have a KCS knowledge article to use, you can use the following example (which we will use in this lab):
Knowledge base: IT
Category: (empty)
Short Description: How do I set the correct date and time on my Mac?
Environment: Mac
Cause: The date or time might need to be set again, or it might be using custom format.
Resolution:
Check Date & Time preferences
1. Choose Apple menu > System Preferences, then click Date & Time.
2. In the Date & Time pane, make sure that “Set date and time automatically” is selected and your Mac is connected to the Internet. Your Mac can then get the current date and time from the network time server selected in the adjacent menu.
To make changes, click the lock to open, then enter your administrator password. If you do not have elevated access, contact IT for assistance.
If you don't have an internet connection, or you want to set the date and time manually, deselect “Set date and time automatically.” You can then click today’s date on the calendar, drag the clock’s hands to the correct time, or use the fields above the calendar and clock to enter the date and time. Then click Save.
4. Note down the number of the KCS knowledge article you are using. For this demonstration, we are using KB0050013. Store this number for use later.
Phase 3: Build a Subflow to collect relevant Knowledge Base categories
1. Navigate to All > Flow Designer. Click New > Subflow.
2. If you have Now Assist for Creator installed, you have the option to build your Subflow with Now Assist. For today, however, we will build it ourselves. Click the Build on your own tab.
3. Name your Subflow: Get IT Knowledge Base Categories. Set the Application to: Global. Click, Build subflow.
4. Click Select to create the inputs & outputs of your Subflow. Click the plus icon next to Outputs.
5. Set the label of your output as Compiled IT KB Categories, then click Done.
6. Open the More Actions menu in the upper right corner of the screen as denoted by the three dots (…), then click Flow Variables.
7. Click the plus icon to add a new flow variable. Set the label to Compiled IT KB Categories as a string, then click Save.
8. Click Add an Action, Flow Logic, or Subflow. Select Action, then type Look up records in the search bar. Click on ServiceNow Core, then finally click on the Look Up Records action.
9. Populate the action as seen below, then click Done (we are populating with the Parent ID’s we collected in phase 2):
- Table: Knowledge Category [kb_category]
- Conditions: Parent ID is a7e8a78bff0221009b20ffffffffff17 OR Parent ID is 9d4737cbff0221009b20ffffffffff33.
10. Click on Flow Logic, then select For Each.
11. Drag and drop the kb_category Records data pill into the Items field, then click Done.
12. Click the small plus icon under the For Each loop and select Flow Logic again. This time, click Set Flow Variables.
13. Click the plus icon, then set the flow variable to Compiled IT KB Categories. Click the script icon next to the text box that appears to make a script field appear. Copy and paste the below code into the area:
var result = fd_data.flow_var.compiled_it_kb_categories
+ fd_data._2__for_each.item.label
+ "\n";
return result;
14. Click Done.
15. Next, add another Flow Logic item, and click Assign Subflow Outputs
16. Set the Subflow output variable to be Compiled IT KB Categories. Drag and drop the flow variable with the same name into the data field. Click Done.
17. Click Save, then Run Test. In the module that appears, click Run Test again.
18. Click on the Assign Subflow Outputs box to review the action that took place. Click on the Runtime Value to see the list of IT Knowledge Base Categories that were returned.
19. Once you validate the categories returned by the Subflow test, return to your Subflow by clicking its name in the tabs at the top. Click Publish in the top right of the screen. If you get a message stating that this is associated to 0 flows, please ignore it and publish it anyway.
🎉 Congratulations! 🎉
We now have all the input data we need to run this custom skill.
Phase 4: Build the custom skill
1. Navigate to All > Now Assist Skill Kit > Home from the filter navigator.
2. Click Create new skill.
3. Name the skill IT KCS Knowledge Categorizer. In this lab, we will be using the Now LLM service to complete our task, so ensure the Default provider and Provider API values are both set to Now LLM Generic. Click Create new skill.
4. Add a new skill input by clicking the plus icon next to it. This is where we add the reference to a knowledge article to use within the prompt.
5. Populate the skill input form with (screenshot below):
Datatype: Record
Table name: kb_knowledge
Name: Knowledge
Choose test record: KB0050014 (the number of the knowledge article we took note of in phase 2).
Once complete, click Add skill input.
6. Next to Tools, click the plus icon.
7. Populate the form as seen below. Note the lack of spaces in the Name, and the resource is the name of our published Subflow, i.e., Get It Knowledge Base Categories.
8. [Optional] Click the Pencil Icon next to the prompt name to rename the prompt. We are renaming it to Knowledge Categorizer here. Click Save changes.
9. In the prompt field, remove the template text and replace it with the following:
You are tasked with classifying input text into one of the following categories:
Each category represents a distinct type of content. Based on the content of the input text,
identify which category best describes it. Ensure that the categorization is based on clear
patterns or keywords in the text that match the defining characteristics of each category.
Instructions:
Review the input text. Choose the category that best matches the content. If the input text
could fit into more than one category, choose the most appropriate one based on the primary
theme.
Example Inputs:
Input: "Title: Can't login to my windows laptop."
Category: Windows
Input: "Issue: Zoom keeps crashing when I start a meeting"
Category: Zoom
Now, classify the following Knowledge Base Article:
Title:
Issue:
Environment:
Cause:
Resolution:
Return only the category. No other text.
10. Click in the prompt field after the string “...one of the following categories:”. Click on Insert inputs, GetITKnowledgeBaseCategories > Compiled IT KB Categories.
11. Repeat the above insert inputs process to add the corresponding text fields to the following fields defined in the prompt:
Title:
Issue:
Environment:
Cause:
Resolution:
Note: because we are using KCS Knowledge Articles, when inserting inputs, ensure you select KCS Article > short description and so on.
12. The completed process prompt should look like the screenshot below. Notice that the Knowledge Base article Short Description field is inserted into the Title section:
You are tasked with classifying input text into one of the following categories:
{{GetITKnowledgeBaseCategories.compiled_it_kb_categories}}
Each category represents a distinct type of content. Based on the content of the input text,
identify which category best describes it. Ensure that the categorization is based on clear
patterns or keywords in the text that match the defining characteristics of each category.
Instructions:
Review the input text. Choose the category that best matches the content. If the input text
could fit into more than one category, choose the most appropriate one based on the primary
theme.
Example Inputs:
Input: "Title: Can't login to my windows laptop."
Category: Windows
Input: "Issue: Zoom keeps crashing when I start a meeting"
Category: Zoom
Now, classify the following Knowledge Base Article:
Title:{{knowledge.ref_kb_template_kcs_article.short_description}}
Issue:{{knowledge.ref_kb_template_kcs_article.kb_issue}}
Environment:{{knowledge.ref_kb_template_kcs_article.kb_environment}}
Cause:{{knowledge.ref_kb_template_kcs_article.kb_cause}}
Resolution:{{knowledge.ref_kb_template_kcs_article.kb_resolution}}
Return only the category. No other text.
13. Click save.
Phase 5: Testing the custom skill
1. Scroll down to the Test prompt section. Click Run tests.
2. Verify that the LLM's output is correct. If you desire, you can iterate upon the provided prompt. However, for our purposes, the skill correctly categorizes the KCS Knowledge article accurately.
3. To verify if information from your skill inputs is being brought in correctly, click on the tab named Grounded prompt. You will see the input that was delivered to the LLM, including the categories and knowledge article text.
4. Once content with the output of your skill, click Finalize prompt.
5. Once content with the output of your skill, click Finalize prompt.
Phase 6: Deploy the custom skill
1. Click on Skill settings at the top of the page.
2. Click on the option named Deployment Settings.
3. Underneath the workflow header, set the value to Other, if not already selected.
4. Tick the option marked UI Action and set the table to Knowledge (to do so, type in kb_knowledge and scroll down until you find it.)
5. Click Create UI Action, then Link to UI Action once it appears.
6. A new window containing your UI Action will appear. Change the value in the Name field to IT KCS Knowledge Categorizer.
7. Additionally, check both the Form button and List banner buttons to have the UI action show up in both a form (i.e., a specific KCS KB article record) and list (i.e., a list of KCS KB article records).
8. In the script field, replace everything from the try statement and below with the code provided below.
try {
// Capture the original category
var originalCategoryGR = new GlideRecord('kb_category');
originalCategoryGR.get(current.kb_category); // Fetch the current category record
var originalCategory = originalCategoryGR.getValue('label'); // Store the original category label
// Execute the request and get the skill's response
var output = sn_one_extend.OneExtendUtil.execute(request)['capabilities'][request.executionRequests[0].capabilityId]['response'];
// Parse the model output to extract category-related information
var modelOutput = JSON.parse(output).model_output;
// Populate the category field using the output of the model
var categorySysId = getCategorySysIdFromModelOutput(modelOutput); // custom function to resolve category from output
if (categorySysId) {
current.kb_category = categorySysId; // Set the category to the resolved sys_id
current.update(); // Save the changes to the KB article
// Inform the user of the original category and the updated category with the record number
gs.addInfoMessage("The category of record " + current.getValue('number') + " has been updated from '" + originalCategory + "' to '" + modelOutput + "'. Please review the update for accuracy.");
} else {
gs.addErrorMessage('No matching category found for the model output for record ' + current.getValue('number') + '. Please review the output or adjust manually.');
}
} catch(e) {
gs.error(e);
gs.addErrorMessage('Something went wrong while executing the skill for record ' + current.getValue('number') + '. Please try again.');
}
// Custom function to get the sys_id of the category from the model output
function getCategorySysIdFromModelOutput(modelOutput) {
// Assuming the model output contains the name or label of the category
var categoryName = modelOutput.trim(); // Adjust based on actual model output structure, trim to remove extra spaces
// Query the KB Category table (kb_category) to get the sys_id of the matching category
var kbCategoryGR = new GlideRecord('kb_category');
kbCategoryGR.addQuery('label', categoryName);
kbCategoryGR.query();
if (kbCategoryGR.next()) {
return kbCategoryGR.getValue('sys_id');
} else {
return null; // Return null if no match is found
}
}
9. Click Update to save the record.
10. Return to your skill within Now Assist Skill Kit.
11. In the module that opens, tick the box containing your finalized prompt, then click Publish.
Phase 7: Activate the custom skill
1. Navigate to Now Assist Admin from within the filter navigator.
2. Click on Now Assist Features, then Other to find your published skill.
3. Click on the tab named Available to find your published skill. Click Activate skill.
4. Set the Display trigger to true, then click Save and continue.
5. Review the next page, then click Activate.
6. Dismiss the message stating confirmation of activation.
Phase 8: See custom skill in action, on a single KCS KB article
1. Type kb_knowledge.list into the filter navigator. Find the knowledge article you wish to run your custom skill on and open the record.
2. Click the UI Action we created – you may have named it something else.
3. Woohoo! Your KCS KB article should be automatically categorized!
Phase 9: See custom skill in action, on a list of KCS KB article
1. Type kb_knowledge.list into the filter navigator.
2. Using the list filter, select Class is KCS and click Run to get a list of only KCS KB articles.
3. Select the articles you wish to run the custom skill on. Click the UI action we just created.
4. Congrats! Your list of KCS KB articles should be automatically categorized!
Future improvement ideas
The purpose of this lab guide is to show you what is possible with NASK. Of course, the skill we built is far from production ready. However, if you intend to continue developing this custom skill, here are some ideas for improvement:
- Additional UI sugar and feedback (e.g., loading screen) when user clicks the UI action button.
- Create a more generalizable skill to be used with any kind of KB article.
- Create additional governance measures, such as getting a human to approve LLM outputs.
- Automatically trigger the NASK skill when any new KB is submitted for review.
Future use case ideas
Building on this demonstration and generalizing the LLM categorization use case pattern, we can extend the Now Assist Skill Kit (NASK) to other areas like IT, SecOps, HRSD, and CSM. Here are some use case ideas to spur your creativity and experiment with NASK. Please let us know if you build any of them!
IT
- Categorization of incidents based on text description data.
- Assignment of requests based on text description data.
IT Operations
- Summarization of alerts, to alert groups, to many alert groups based on description and other data.
- Categorization of these summarizations to assign work to the right team.
Security
- Categorization of security events, incidents, vulnerabilities based on text description data.
Customer Service
- Assignment of customer case to correct customer service or another team.
HR
- Assignment of HR case to correct team.
- 7,875 Views
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
I appreciate this walkthrough @johlee !
In the example use case for HR, what you listed sounds similar to predictive intelligence. In the context of NowAssist, is the suggestion for a use case where we have NowAssist identify categories/assignment groups of cases based off a given request? Or something else?
Thank you again!
- Joey
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Hi @JT8,
The example use case for HR would be building a Now Assist custom skill to handle categories/assignment using the unstructured text contained in the case.
You bring up a great point though, some categorization tasks may be better suited for classic (non-LLM based) ML techniques vs. LLM-based methods. However, I've found that when it comes jobs to be done that require synthesizing large amount of unstructured text into a decision, LLMs (Now Assist) really shines.
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Can this solution be scaled, to say review 800,000 records?
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
@Jayden4, yes I see no reason why this custom skill couldn't be scaled out to many records. I say this because the categorization of each record is an independent call to the LLM which reduces the risk of overfilling the LLM context window with too many tokens resulting in hallucination. However, if the unstructured text in each KB is vast and you have many categories and sub-categories, you may still run into this challenge, so definitely test this skill out on a few large KBs and check the veracity of the output before proceeding.
Continuing, the only thing I would change would be to adjust the skill so that it is called in a flow, with logic to automatically iterate over the subset of records you wish to apply the skill on. You may want to add a sleep or wait for duration condition after each iteration as to not overload the system/run it during a less busy part of the day.
At the same time, I would also recommend testing this over a smaller batch of records before doing full run.
Let me know if this helps and if you have any further questions!
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
Thanks for the info @johlee, I built some use cases that also used a UI Action in the list view to review and apply tags, additionally to updating fields based on the LLM response, however as it was performed via a UI action, you could only perform the action for 300 seconds before the system would stop the transaction.
Any ideas on how to get around the 300 second timeout would be appreciated!
- Mark as Read
- Mark as New
- Bookmark
- Permalink
- Report Inappropriate Content
@Jayden4 check out the sys_one_extend_rate_limit_rules table (search 'sys_one_extend_rate_limit_rules.list' in your filter navigator)
I believe that timeout makes sense with the default rate limits.
Also, I would caution you on your use case(s) and double-check any entitlements you have! (But this may be something your ServiceNow PoC may need to confirm)