How to Downlaod large count of records in a single file

[SivaprasadV]

In servicenow I have created a custom table. User insterted more than 10 lakh line items. Now I want to download all the records in one file. How can I do that.

[pr8172510]

Hi @SivaprasadV,

For 10+ lakh  records, the List Export UI is not recommended due to export limits, browser timeouts, and memory constraints.

1. Table API with Pagination
   • Use the Table API with sysparm_limit and sysparm_offset (or keyset pagination using sys_id) to retrieve records in batches.
   • This is the preferred approach for large-scale data extraction.
2. Data Export / Integration Tools
   • For recurring exports, use IntegrationHub, JDBC, ETL tools, or data integration platforms rather than UI exports.
3. Scheduled Server-Side Processing
   • If a file is required, generate it in batches using a Scheduled Script Execution and attach/store the output.

• Increasing properties such as glide.csv.export.limit or glide.ui.export.limit is generally not recommended for exporting 1M+ records, as it can impact instance performance.
• Exporting 1 million records in a single CSV file may also hit attachment and memory limits. Consider splitting the export into multiple files.

For a one-time export, Table API pagination is typically the safest and most scalable approach according to ServiceNow best practices.

[SivaprasadV]

Hello  @pr8172510 ,

But the client wants to download all records in a single file. they don't want to download it in batches wise. and I tried modifying the properties which you shared. and it started exporting all of the sudden after exporting 35k records it stopped exporting. 

[Tanushree Maiti]

Hi @SivaprasadV 

1. 

• Navigate to System Export Sets > Export Definitions and click New.
• Set the Table to your custom table and select the fields you want to export.
• Go to System Export Sets > Create Export Set.
• Select your Export Definition, choose your target MID Server, and set the file format to CSV.
• Click Export Now. The MID Server will generate the complete file in your specified network directory

Refer:https://www.youtube.com/watch?v=qbka771D2Jg&t=199s

2)Using API

Refer:

https://www.servicenow.com/community/developer-forum/how-to-export-more-than-10000-records-in-servic...

3) Run an Asynchronous Background Script 

Refer: https://www.servicenow.com/community/developer-forum/how-to-export-more-than-10000-records-in-servic...

[vaishali231]

Hey @SivaprasadV 

Exporting 10+ lakh (1 million+) records in a single file using the standard ServiceNow List Export functionality can be challenging, even after increasing the export-related system properties.

Since your export starts successfully but stops after approximately 35,000 records, it is likely encountering one of the platform limitations, such as:

1. Transaction timeout
2. JVM memory (heap) constraints
3. Node resource limits
4. HTTP response size limitations
5. Attachment size restrictions (depending on the export method)

Increasing the export properties alone does not guarantee that very large exports will complete successfully.

For datasets of this size, the recommended approach is to avoid browser-based exports and instead use a server-side or external export process, for example:

1. Create a Scheduled Script Execution that processes the records in batches and generates the CSV in the background.
2. Generate the CSV server-side and attach it to a record for the user to download once the job completes.
3. Use Export Sets, the Table REST API, or a MID Server/external ETL tool to extract the data and generate the final file outside the UI.
4. If the requirement is for reporting or analytics, consider exporting from a reporting/replica database instead of the production instance.

*************************************************************************************************************************************

If this response helps, please mark it as Accept as Solution and Helpful.

Doing so helps others in the community and encourages me to keep contributing.

Regards

Vaishali Singh

Servicenow Developer
Linkedin - https://www.linkedin.com/in/vaishali-singh-2273361bb