InfoSec Global integration crawling in PROD but very efficient in DEV & Stage

aislingo
Tera Contributor

4 weeks ago

Something is affecting the speed of our ISG integration in PROD.

 

I have doubled the data sources and templates.

DEV = Zurich version, plug in 1.2.5

STAGE = Yokohama version 1.2.2

PROD = Yokohama version 1.2.2

 

DEV & Stage are fine. PROD is slow as hell, taking 3 days to process.

Error: Job exceeded processing time and was forced to complete status.

Any ideas? 

0 Helpfuls
  • 363 Views
4 REPLIES 4

Me Being Mustaq
Tera Guru

4 weeks ago

Hi @aislingo ,

 

Greetings for the Day!!

 

The severe slowdown and processing timeout of your ISG integration job in PROD, compared to faster DEV and STAGE environments, despite similar Yokohama versions and plugin levels, indicates environmental or data-specific issues causing the performance.

 

Possible Causes:

  • PROD likely has much larger volumes of data and more complex dependencies than DEV or STAGE.

  • Increased data can exponentially increase processing time, especially if the data sources and templates have grown.

  • PROD servers might have stricter resource limits or slower performance (CPU, memory, I/O) than DEV/STAGE.

  • Review instance performance monitoring metrics to check for CPU spikes, memory contention, or database locks during the job.

  • Multiple jobs or integrations running concurrently in PROD could be causing contention.

  • Check if other heavy batch jobs run at the same time impacting resource availability.

  • This specific error means the integration job ran longer than the max allowed time, so ServiceNow forcibly ended it.

  • Investigate job logs for where the job is spending time—possibly in transforms, lookup, or script-heavy steps.

  • Confirm that PROD data sources and templates are identical in logic and configuration, not just count.

  • Complex templates or dependencies can cause inefficient processing.

  • Large data volumes require proper indexing in the underlying tables.

  • Check for missing or fragmented indexes that can slow queries in PROD.

  • Enable debug logging or telemetry for ISG integration jobs to pinpoint bottlenecks.

  • Compare query times, transform times, and staging vs. production differences.

Suggested Actions to Resolve

  • Reduce Job Batch Size:
    If configurable, reduce the batch/work chunk size to allow jobs to complete within time limits.

  • Optimize Templates and Transforms:
    Review and simplify templates or import set transforms consuming excessive time.

  • Add or Rebuild Database Indexes:
    Ensure frequently queried columns are indexed on tables used by ISG.

  • Stagger Job Runs:
    Schedule heavy jobs at different times to reduce concurrent load.

  • Scale or Upgrade PROD Resources:
    If possible, increase CPU/RAM allocated to PROD instance or database servers.

  • Contact ServiceNow Support:
    If performance issues persist, provide job logs and environment info to ServiceNow for deeper platform analysis.

If it is helpful, please hit the thumbs button and accept the correct solution by referring to this solution in the future it will be helpful to them.

 

Thanks & Regards,

Mohammed Mustaq Shaik

aislingo
Tera Contributor

3 weeks ago

Hi Mohammed! Thanks for your response. 

 

I have tried these 2 steps below and seen some improvement but some processes are still failing

 

  • Optimize Templates and Transforms:
    Review and simplify templates or import set transforms consuming excessive time.

  • Stagger Job Runs:
    Schedule heavy jobs at different times to reduce concurrent load.

Are there any official KBs for achieving chunk size reduction?

 

  • Reduce Job Batch Size:
    If configurable, reduce the batch/work chunk size to allow jobs to complete within time limits.

 

0 Helpfuls

Me Being Mustaq
Tera Guru
In response to aislingo

3 weeks ago

Hi @aislingo ,
 
Yes, ServiceNow has several official KBs and documentation entries that cover batch size tuning and chunk-size optimization for integrations and large data jobs. These guides provide safe, platform-supported methods to reduce batch or work chunk size so that processing heavy jobs (like your ISG integration) completes within time limits.
 

Relevant Official References

  1. KB0787516 – Job exceeded processing time
    This KB explicitly addresses integrations such as Rapid7 or Qualys that fail with the “Job exceeded processing time and was forced to complete status” error.
    Recommended resolution steps include reducing Rows Per Request to somewhere between 20K–50K per cycle (default is often 100K).
    Path example: Rapid7 Vulnerability Integration → Integrations → Add Rows Per Request column → Adjust values accordingly.​
  2. KB0869409 – Optimizing a Query Builder query via batch size
    Introduced a Batch Size per Query mechanism where admins can fine-tune record retrieval limits for large queries. It covers how to:
    • Set global property glide.cmdb.query.batch.size
    • Override this per-query by editing the Execution Batch Size field in qb_saved_query records.
      Smaller batch values reduce system load and improve query completion time for long-running jobs.​
  3. KB0863634 – Performance tuning tips for Qualys Import Jobs
    Though this applies to vulnerability imports, the pattern is similar.
    It notes lowering the pagination truncation limit and batch evaluation time to prevent integrations from overloading nodes during large data pulls.​
  4. Integration Commons and Partition Sizing (Zurich Documentation)
    The Zurich release docs add parallel load features where partition or chunk size is auto-computed or tunable in connector configurations (e.g., Integration Commons for CMDB → partition size computation).
    Adjusting chunk sizes within these parallel frameworks prevents job timeouts under heavy load

Recommended Next Steps for Your Case

  • Check the ISG data import scheduled job configuration and reduce Rows Per Request to below 50K.
  • Review any Query Builder or CMDB Query Batch Size properties, especially if the integration leverages them indirectly through transforms.
  • If using Integration Commons or Service Graph Connectors, inspect the partition size configuration as suggested in Zurich docs.
  • Validate job concurrency settings so that your staggered scheduling aligns with reduced chunk sizes.
Applying these supported tuning settings typically stabilizes long-running integrations across environments while keeping memory use manageable.

Please appreciate my efforts, help and support extended to you by clicking on – “Accept as Solution”; button under my answer. It will motivate me to help others as well.

 

Thanks & Regards,

Mohammed Mustaq Shaik

 

 

Me Being Mustaq
Tera Guru
In response to Me Being Mustaq

3 weeks ago

Hi @aislingo ,


Hello

Greetings!

 

you may find below thread helpful:

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0869409
https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0787516
https://support.servicenow.com/nav_to.do?uri=%2F$pa_dashboard.do%3Fsysparm_dashboard%3D6e2ea28ddbe9a...
https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0863634

 

Please appreciate my efforts, help and support extended to you by clicking on – “Accept as Solution”; button under my answer. It will motivate me to help others as well.

 

Kind Regards,

Shaik Mohammed Mustaq