Recently, i started working on Architecting a few custom AI Skills and Agents using ServiceNow's Now Assist platform, While i development experience was smooth, I quickly hit a critical question. What does good test data actually look like for AI.
Building the skill logic was only half the job, but the real challenges was testing it properly with relevant data set. Without a Quality or realistic data, it is very hard to determine if the skill was truly ready for production movement.
That's when i started exploring the "Now Assist Data Kit" . NADK , Shortly, is a powerful tool that helps you define, organize and manage the kind of test data your skill truly need to learn.
before getting into this NADK, Why good test data Matter for AI ?
Evaluating custom skill without the right data is like testing a car without a test drive on road. the quality of the input data directly impacts the reliability of your evaluation results and skills.
Then What Makes a Good Test Dataset?
Below are six simple things that makes a test data set great.
01. Ground Truth Data - Ground truth is the correct, expected output used to measure how accurately an AI skill performs during the testing. This data includes benchmark or "Correct" outputs for comparison, enabling you to measure how closely the skill's response aligns with ideal outcomes.
02. Accurate Data - Your data should contains validated entries. The erroneous in input can mislead skill performance and introduce false positives/Negatives or hallucinations.
03 Right Sized Data - Large enough to yield meaningful insights, but manageable set of data is enough to maintain, interpret and iterate quickly.
04. Unbiased Data - Data set contains balanced representation across data categories, avoiding skewed outputs due to over presented data types or perspective, so unbiased data plays a major role in the output
05. Realistic Data - Data should reflect actual enterprise data, should not be overly synthetic, which makes the evaluation process more relevant and actionable.
06. Diverse Data - your data should cover a wide range of scenarios, including common and edge cases, to ensure the skill performs under varied inputs and real word scenarios.
now that you understand what type of data is required it make your AI to provide optimal results, the next steps is making that data easy to collect , manage and use. That's exactly this "Now Assist Data Kit" comes in place.
In the Now Assist Data kit , we are collecting the real world tasks that AI should handle, Adding their ground truth response and grouping similar examples into collections for structured testing and sharing those collection to AI skill kit for execution.
In the next part, I will through how to build you Data kit user Journey and your first data set using Data kit.
Reference :
1. https://www.youtube.com/watch?v=CCmjsa2DmKo
2. https://www.servicenow.com/community/now-assist-articles/now-assist-data-kit-faq/ta-p/3105073
5 Comments
