Qualys Vulnerability Import - Job exceeded processing time and was forced to complete status

Sohaib Sarwar1 · ‎10-16-2020

Qualys host detection daily job completing with Substate status Failed. For past couple of days our daily host detection job is not able to ingest all of Qualys data into Snow Vulnerability Response module. Qualys Import Source Substate failed comes back with message under Notes "At least one import queue entry is in error. No more data to process at this time". When you drill down Qualys import source and look at the VINTPXXXXX number the message in Notes states "Some import queue entries are in error". After drilling down the VINTPXXXXX number the Qualys Vulnerability Import has status Error and with processing notes message with "Job exceeded processing time and was forced to complete status". This issue started happening in our prod instance without any recent changes to prod instance. Can someone advice what could be causing the some of the Qualys import to fail? I have also attached screenshot for reference.

andy_ojha · ‎10-16-2020

Hey there,

Typically when this happens, you are bumping up on the preset 3600s (60m) timeout threshold on the import queue -> (time to process the individual paginated payload files sent to ServiceNow).

Currently, this threshold is hardcoded in the core script include "VulnerabilityDSAttachmentManager" and is not configurable without customizing the script include.

By chance - was this your very first import from Qualys (for detection data)? Or was this from a delta data load (import since, set to yesterday), or a backfill data load (where you set the Import Since a days or weeks ago)?

In some cases, these files can grow to be very large (especially if we are using Qualys Agents) - where a given host from Qualys, is returned with a large amount of data to be processed (vulnerability detections).

Your best bet is to open a HI Support Ticket - they can review the files returned from Qualys to assess if they are too large to be processed in the 60m window - and provide you some guidance on tuning for your specific situation in order to move forward.

View solution in original post

Sohaib Sarwar1 · ‎10-21-2020

We test it in our UAT by bumping the timeout threshold on the import queue from 60 - 90 and was successful. We also pushed the change to prod yesterday and imported data back to 9/27 without any Substate status Failed. This suggestion was also given by the HI Support team. Thank you

Salma Laskar · ‎12-03-2024

Hi Sohaib,

While analyzing the similar issue, observed in failed run -
import queue entry error - 'Job exceeded processing time and was forced to complete status.'
You can refer : Qualys host detection Integration for Security Operations failing - Support and Troubleshooting

this value could be found under -

store application [sys_store_app] table

filtered with name "Qualys Integration for Security Operations"

under customized files tab in the related list, search with display value "truncation_limit"
you have to lower the value from default 500 to 300 apparently.

aAkb