Some PDIs are currently unavailable, and PDI actions are paused. View the latest updates here. Read More

Now Assist Skill - Optimize and Evaluate Section Loads Indefinitely When Maximum Records > 1

BhavaniYamsani
Tera Expert

3 weeks ago

Hello Community,

I am experiencing an issue while creating and evaluating a custom skill in Now Assist Skill Studio.

Environment

  • Module: Now Assist Skill Studio
  • Skill Type: Custom Skill
  • Step: Optimize and Evaluate
  • Dataset Source: Table (sn_si_incident)
  • Use Case: Major Security Incident Qualifier Check

Issue Description

When I create a dataset in the Optimize and Evaluate section and set the Maximum Records value to more than 1, the evaluation/optimization process keeps loading indefinitely and never completes.

However:

  • When Maximum Records = 1, the process works as expected.
  • When Maximum Records > 1 (for example, 2 or more), the page continues loading forever and no results are returned.
  • No clear error message is displayed in the UI.

Steps to Reproduce

  1. Open a custom skill in Now Assist Skill Studio.
  2. Navigate to Optimize and Evaluate.
  3. Create a dataset using a table (e.g., sn_si_incident).
  4. Set Maximum Records to a value greater than 1.
  5. Save or run the evaluation/optimization.
  6. Observe that the process remains in a loading state indefinitely.

Additional Information

  • The dataset is successfully created.
  • The issue occurs consistently whenever the record count exceeds 1.
  • Tested with different record values and observed the same behavior.

Questions

  1. Has anyone encountered this issue in Now Assist Skill Studio?
  2. Are there known limitations or prerequisites when evaluating datasets with multiple records?
  3. Are there specific logs, system properties, or background jobs that should be checked for troubleshooting?
  4. Is this a known product issue in any ServiceNow release?

Any guidance or troubleshooting suggestions would be greatly appreciated.

Thank you!

Regards
Yamsani Bhavani
ServiceNow Developer - SecOps, IRM, Custom Applications, AI Now Assist

Preview file
108 KB
0 Helpfuls
  • 518 Views
1 REPLY 1

Vikram Reddy
Tera Guru

2 weeks ago

Hi @BhavaniYamsani,

 

My honest read: this looks like a defect, not a config mistake on your end. An evaluation run against a two-record dataset should finish in well under a minute, not spin forever with zero error surfaced anywhere in the UI. Before you log a case though, there are a couple of very checkable things worth ruling out first.

The Optimize and Evaluate tab bundles two different capabilities under one screen: a plain Evaluate run, and Automated Prompt Optimization (APO). That combined framing showed up when APO landed in Skill Kit 4.0.3 (Xanadu Patch 9 / Yokohama Patch 3). APO specifically calls for a dataset of at least 100 records to do anything meaningful, it needs that volume to actually optimize against. Your MSIQC-Demo dataset is nowhere near that. If bumping Maximum Records from 1 to 2 is enough to route you from a trivial single-record test into the "real" optimization path, that path could be sitting there waiting on a precondition it will never satisfy, with nothing telling you why. Worth confirming exactly which mode you're triggering, Evaluate only, or Optimize, and how many total records actually exist in that dataset versus what Maximum Records is capping.

Second thing: sn_si_incident carries tighter row-level security than a normal incident. Security Incident records are only visible to users holding the right sn_si role (sn_si.basic, sn_si.analyst, sn_si.admin, etc.), and admins can further restrict visibility to specific assignment groups via the "Enforce restriction" setting. It's plausible the run is stalling while trying to pull field data for a record that the execution context can't fully see, and that failure never makes it back to the modal as a visible error. An easy isolation test: point the same dataset config at the plain incident table instead and set Maximum Records to 2 there. If that completes cleanly, the problem is data-visibility-specific to sn_si_incident, not a hard bug in the Maximum Records field itself.

Also don't trust the modal spinner as your only signal. Go into the Automated Evaluations or Optimization runs node in the left nav, you already have MSIQC-E listed there, and check that run record's actual status directly. It's not unusual for the backend job to finish or error out while the frontend polling never resolves. Worth a glance at your instance's Now Assist consumption too, since evaluation burns one Assist per record processed, and a quota problem can produce a silent stall instead of a clean error message.

If none of that explains it, this is worth escalating to Now Support rather than digging further blind. A 100% reproducible hang tied to one specific field value, with zero error, is exactly the profile of something that turns into a PRB once support can see the backend job logs that aren't visible to us as admins.

Happy to dig further if you can confirm which mode you're actually running and the total record count behind that dataset.

 

Thank you,
Vikram Karety
Octigo Solutions INC

0 Helpfuls