Data Source properties

Kenneth3
Tera Guru

‎06-16-2022 12:46 AM

Hi,

I am importing CMDB data from a third party source with IH-ETL (IntegrationHub ETL). I have created a REST (IntegrationHub) Data Source with JSON format.

In the Data Source, I have enabled the 'Use Batch Import' and set the Batch Size to 2.000 records. But when running the import, only 1.000 records are being fetched. Total import is 40.000 records.

One thing I want to try is to modify the properties glide.import_set_load_usebatch and glide.import_set_load_batch_size, but I can't find them. Does anyone knows where I can find those?

And if the Batch Size is 1.000 records, I would expect to see 40 Import Sets with each 1.000 records, but I see only one Import Set with 1.000 records. Does anyone had the same experience with this and knows how to deal with this?

KR,
Kenneth

find_real_file.png

Preview file
63 KB
0 Helpfuls
  • 1,598 Views
1 REPLY 1

Shreya Shah
ServiceNow Employee
ServiceNow Employee

‎06-29-2022 12:48 AM

Try creating those properties in the System Properties [sys_properties] table and setting values that you need as threshold. You can open that by typing "sys_properties.LIST" in the navigator.

Per documentation (https://docs.servicenow.com/en-US/bundle/sandiego-platform-administration/page/administer/import-sets/reference/r_DataSourceFileTypeFields.html) the order in which it checks is "The priority order for the properties and fields are as follows: If the property is set, use the property value. If not, use the field value. If neither property is set, use the default batch size of 1000." From your screenshot the batch size for this import is 1000 and the default is 1000 too in case the property is not set.

0 Helpfuls