What is ground Truth in Now Assist Data kit ?

[Thulasidharan]

Hi Everyone,
     I was working on the Now Assist custom skill, I created the custom skill to create the summary for the capability. Now I need to evaluate the custom skill output using the data kit.  I created the data set in the Data kit, There is an option called create ground truth.

I checked the Data kit FAQ, it mentioned as '

What is Ground Truth, and how is it useful?

Ground Truth is the desired output from a generative AI action for a given record. This is used during evaluation when comparing the output of the LLM to the ground truth to determine if the response was successful according to a particular measure.'


I just want to know what should put here, I seen an video included in the FAQ which they

mentioned to give one word for categorization. I want to know how this works

Should I include the output for the capability which is expected?

Should I include the expected output from the custom skill for each record?

Can anyone help on this, what should I gave it to the ground truth?
Thanks🙂

[PaulSylo]

Hi @Thulasidharan 

Ground truth is the expected output given by reliable source,

For example your input is - " i cant log into my email" , Ground truth : Access 2. My laptop screen is blank: Ground truth will be Hardware. Basically, we are calcifying the single consistent label for training the data you have created in the data kit.  so you try Should I include the output for the capability which is expected?

[Thulasidharan]

Hi @PaulSylo 
So it means that we are giving labeling for the output, right?
If it is then I want to know that how it can check the accuracy of the output?
Let's say, I'm creating summary for the capabilities/ ServiceNow products, then what should I mention in the ground truth for each record? 

Is that an one word label? Or I can go with an expected summary in the grounded truth?

For an example, I'm asking create summary about ServiceNow ITSM, and I'm giving all the inputs to the custom skill and also to evaluate this I create the dataset with same input and what should I put in grounded truth?
A label or expected summary like paragraph about the ITSM? 

[PaulSylo · Accepted solution]

HI @Thulasidharan  - if your custom skill's capability is summarization, then it is near truth paragraph ! if your skill is summarization ( generating a paragraph of text) then Ai would learn the output a label, not a summary