Moving SN archived data to cloud storage with best approach

[dsc123]

Hi Gems,

We have very large amount(10TB) of table data(incident, work orders, change, RITMs, catalog tasks, etc.). in servicnow and we have done archive of those tables and Now we have to move those archived table data to some cloud service storage. In order to move these data what is the best approach, can some one guide us over here. I do know with Rest(Table) API it has some limitations like it can allow only 10000 records per get and does not include referenced tables.

[Ratnakar7]

Hi @dsc123 ,

Moving archived data from ServiceNow to a cloud storage service like AWS S3, Azure Blob Storage, or Google Cloud Storage can be done efficiently with the right approach. Here are the steps and considerations for this process:

1. Export Data from ServiceNow:

   • Use ServiceNow's built-in export options to extract the archived data from your tables. This could include using list views, reports, or scripted exports depending on your data size and complexity.

2. Data Transformation (Optional):

   • Depending on your requirements, you might need to transform the data into a format compatible with your chosen cloud storage service. For example, you might need to convert data to CSV, JSON, or other formats.

3. Split Data into Smaller Chunks:

   • As you've mentioned, REST API has limitations in terms of the number of records per request. Split your data into smaller chunks or batches for more manageable processing.

4. Select Cloud Storage Service:

   • Choose a cloud storage service that suits your organization's needs. AWS S3, Azure Blob Storage, and Google Cloud Storage are popular options. Evaluate pricing, performance, and features to make an informed decision.

5. Transfer Data to Cloud Storage:

   • Use the cloud storage service's native tools or SDKs to upload your data. Each cloud provider offers tools like the AWS Command Line Interface (CLI), Azure Storage Explorer, or Google Cloud Storage Console.

6. Optimize for Parallel Processing:

   • To speed up the data transfer, consider parallelizing the uploads. Many cloud storage services allow parallel uploads of objects. You can also use cloud-native services for large-scale data transfer, like AWS Snowball or Azure Data Box.

7. Manage Security and Access Control:

   • Implement appropriate security measures, such as encryption and access control, to protect your data. Most cloud storage services provide robust security features.

8. Verify Data Integrity:

   • After transferring data, verify its integrity by comparing checksums or performing data validation checks.

9. Data Catalog and Metadata:

   • Maintain a catalog of the data moved to cloud storage. Include metadata like creation date, source, and any other relevant information. This will help in data governance and future retrieval.

10. Cost Monitoring and Optimization:

    • Keep an eye on the costs associated with storing data in the cloud. Implement data lifecycle policies to automatically archive or delete data that is no longer needed.

11. Backup and Disaster Recovery:

    • Ensure you have backup and disaster recovery strategies in place for data stored in the cloud.

12. Documentation and Training:

    • Document the entire process, including the steps taken, tools used, and any custom scripts or configurations. Provide training to the team responsible for maintaining the data in the cloud.

13. Testing and Validation:

    • Perform thorough testing and validation of the data after it's in the cloud storage to ensure it can be retrieved and used effectively.

14. Monitoring and Maintenance:

    • Set up monitoring to detect any issues with data storage and access. Implement regular maintenance tasks as needed.

15. Archiving Data in ServiceNow:

    • Once data is successfully moved to cloud storage, you may consider archiving or deleting it from ServiceNow to free up database resources. Ensure compliance with data retention policies.

Thanks,

Ratnakar

[dsc123]

Thanks for your detailed approach @rat , When you say with smaller chunks approach getting 10 TB of data may take in days and The approach what I am seeing currently is Table API approach(https://docs.servicenow.com/bundle/utah-api-reference/page/integrate/inbound-rest/concept/c_TableAPI...) , where in its not supporting compress techniques and limited to only 10K records does not supports referenced data, DO you any other approach where Lets say If I get incident table will i get all referenced data?

[dsc123]

Any thoughts on this or anyone implemented above use case?