Qualys Jobs time out after a while when pulling historical data

IceIronDragon
Tera Guru

Hello All,

We have a situation where in we have integrated Configuration Compliance with Qualys on OOB parameters.

It works fine when we pull in present data.

The job times out when historical data is being pulled.

When we put start time more than a month or 90 days data.

We raised it with Qualys and qualys confirmed that there is no issue at there end.

Has anybody faced this kind of issue and what was done to mitigate it.

2 REPLIES 2

andy_ojha
ServiceNow Employee
ServiceNow Employee

Hey there,

When you say the job times out - are you getting the error message that a particular integration run exceeded the 60min threshold for processing?

You should open a ServiceNow Support Ticket to really get the right eyes on the problem here as there could be a few different things happening.

The first exercise would be to first analyze how much time each file is taking to process -- and the particular flavor of import you are doing - first initial import or subsequent daily deltas...

Consider when a new Qualys Result / Posture is brought in - it has to go through a lot of processing with the core CC logic (Assignment, Scoring, etc)...  Whereas, a Qualys Result that has already been brought in, is brought in again with an update - it does not have to go through the same processing - as in, the first import is much heavier than the delta imports.

One tidbit you can try in the meantime, on the actual REST message (LIST) for the Qualys Result Job - on the HTTP Query Parameters tab, you'll see the 'Truncation Limit` set to 5000 ... which relates to the pagination ... as in, the amount of data per file (total count of compliance posture entries)...  There's no magic number but you can try to reduce that down iteratively.

It may prove worthwhile to reduce the pagination for the very first big import as a series of processing is expected to occur for net new data - as opposed to updates and smaller sets of net new data in delta imports later on - as in drop it below 5000 for the first big import if you really absolutely need to go back more than 7d for the first import - and then put it back to 5000 after for the deltas.

Totally understand the need to pre-seed the NOW CC side with outstanding compliance issues (the FAILED Results).  Unfortunately, the integration is not using the Time Filtering param like -> 'Evaluation Date` ... 

Because the integration is using the the 'Status Change Since', it requires going back in time quite a way to ensure you have the full set of FAILED Results to start with - because some FAILED results may have had their Status changed many weeks / months ago.

Reference - Qualys Time Filtering for Posture API

find_real_file.png

Reference - Qualys Pagination for Posture API

 

find_real_file.png

find_real_file.png

Shivam Sarawagi
ServiceNow Employee
ServiceNow Employee

Hi,

 

Since you mentioned that the run substate is time out check the "Data retrieval duration" of the integration processes present for this run. You may find one of the processes taking more than an hour. This means Qualys API took more than an hour to respond.