- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2025 08:37 AM
Hi Everyone,
I was working on the Now Assist custom skill, I created the custom skill to create the summary for the capability. Now I need to evaluate the custom skill output using the data kit. I created the data set in the Data kit, There is an option called create ground truth.
I checked the Data kit FAQ, it mentioned as '
What is Ground Truth, and how is it useful?
Ground Truth is the desired output from a generative AI action for a given record. This is used during evaluation when comparing the output of the LLM to the ground truth to determine if the response was successful according to a particular measure.'
I just want to know what should put here, I seen an video included in the FAQ which they
mentioned to give one word for categorization. I want to know how this works
Should I include the output for the capability which is expected?
Should I include the expected output from the custom skill for each record?
Can anyone help on this, what should I gave it to the ground truth?
Thanks🙂
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2025 12:16 PM
HI @Thulasidharan - if your custom skill's capability is summarization, then it is near truth paragraph ! if your skill is summarization ( generating a paragraph of text) then Ai would learn the output a label, not a summary
PaulSylo
Kindly mark "helpful", if this helps, or Mark as "Accepted " if it solves your issues !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2025 10:20 AM
Ground truth is the expected output given by reliable source,
For example your input is - " i cant log into my email" , Ground truth : Access 2. My laptop screen is blank: Ground truth will be Hardware. Basically, we are calcifying the single consistent label for training the data you have created in the data kit. so you try Should I include the output for the capability which is expected?
PaulSylo
Kindly mark "helpful", if this helps, or Mark as "Accepted " if it solves your issues !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2025 11:08 AM
Hi @PaulSylo
So it means that we are giving labeling for the output, right?
If it is then I want to know that how it can check the accuracy of the output?
Let's say, I'm creating summary for the capabilities/ ServiceNow products, then what should I mention in the ground truth for each record?
Is that an one word label? Or I can go with an expected summary in the grounded truth?
For an example, I'm asking create summary about ServiceNow ITSM, and I'm giving all the inputs to the custom skill and also to evaluate this I create the dataset with same input and what should I put in grounded truth?
A label or expected summary like paragraph about the ITSM?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-13-2025 12:16 PM
HI @Thulasidharan - if your custom skill's capability is summarization, then it is near truth paragraph ! if your skill is summarization ( generating a paragraph of text) then Ai would learn the output a label, not a summary
PaulSylo
Kindly mark "helpful", if this helps, or Mark as "Accepted " if it solves your issues !