What is the best practice for setting up data archiving and data deletion?

[Chas]

We are looking to set up data archiving and data deletion across all our instances and I am seeking some best practices and ideas from other Service Now users. Below are some considerations we would like to adopt in any policies:

• Cloning instances - how much data is actually needed in sub-prod instances? Can we setup 30 days of data during a clone, or does each instance need its own set of delete policies each time a clone is performed?
• Compliance to FCA and other data legislations (Rather than keeping data for 7 years, can we archive most of the 7 years? Or does it need to be kept unarchived?)
• Mapping all of the 7,000+ tables into a Service Now archiving/delete policy. How is this best achieved? Can archiving groups be setup and then have tables assigned to a group? Is there an easy way to see storage consumption by table/package etc? Does archived data consume less storage than retained unarchived data?
• Process to automatically add new tables resulting from new apps/upgrades into a archive group. What is the best way to identify new tables being installed during an upgrade etc?

Any other good practices and ideas would be welcome.

Thanks.

Chaz.

[bammar · Accepted solution]

Hi Chaz,

 I have been an admin/developer for 8 years and I cant really say best practices but i follow a pragmatic approach.  80% of all results only require 20 % effort. No use putting energy to things that have low utility and low benefit.

• Archiving - Candidates for archiving are - Tables like incidents that grow beyond 1 million records. Performance on queries starts to suffer, loading lists, scripts etc suffers. Consider in instance 5 years old and Incident is now surpassing 1 million records- The organization most likely already reported on everything they need to for years 1 and 2 already- do they really need to see the simple pw reset incidents from 5 yrs ago? I think for the 7 Yr standard question- it doesnt matter where you keep it as ong as you keep it- think about it- Some companies dont have ServiceNow , they have spreadsheets or worse PAPER! so being in ServiceNow as an archive as opposed to in the live table is semantics.
• Methods - Work with certain Groups who worked the tickets and set up archiving policy appropriately = Maybe the group that does PW resets will allow a policy of archiving any PW reset incidents 6 months or older- while another group say Legal will say - we only get 100 tickets a yr and we need ours in Prod- you leave those in there. So basically you find practical groups of tickets and timeframes worked out with the stakeholders and you set it and ServiceNow autoarchives.
• Deletion - I would NEVER delete anything unless i really had to. I think Archive is the savior for those who dont want to delete- Archiving creates an another table that correspponds to the original and stores the archived records there- I never detected latency and ServiceNow never said hey you have too many records and our storage cant handle it or something like that. Deletion also goes against record keeping standards for say Medical or Schools. 
• Tables- Focus on Archiving records from tables with alot of records-  If say the Demand Table only has 600 records after 8 years there is no real need to archive- ideally we can search all as needed from that table. I wouldnt even think of archiving 98 percent of the tables in ServiceNow- some system tables have auto- archiviing/deletion processes and i leave those alone...
• If you totally remodel a customized application or area- you could archive the old records as part of that project
• Auditing- this may necessitate the need to fetch records from the past so only Delete if you have no choice in my opinion
• Clones- These are already streamlined in terms of the data- for instance by default Clones dont copy attachments, they also I beleive have 90 days of records and you can change that . I have no care in the world how many GIGs my instances hold on ServiceNows Servers haha As long as Customers are happy, ServiceNow hosts are happy, I am happy

[Chas]

Thanks for the comprehensive answer. We have found that Service Now does allocate storage limits, and as we have been using Service Now since 2010 storage consumption across instances has grown. I have a few additional questions:

1. Given the storage limitation, would archiving data consume the same amount of data as it being unarchived? ie. I'm trying to establish if I want to lower the storage, do I have to delete the oldest data, or will archiving bring it down?

2. Is there a way of working out what objects in Service Now are consuming the most storage? I imagine emails, attachments and images will be the largest consumer. A neat way of tackling the largest culprits will be good 🙂

3. How does archiving data affect performance analytics? Could year on year trends still be seen, if data has been archived?

Any other suggestions or useful tips would be great from any company.

Thanks.

[bammar]

1. I dont beleive archiving lowers the amount of data. It is merely a transfer of records from one table to another.

2. I think attachments would be the culprit in terms of space.

3. I think archiving data from tables that are used in performance analytics may cut those data points out of the results.. You can experiment with that one day in sub prod  and wait one evening to see- as performance analytics runs once a day.

Finally- Depending on what you have to when your storage limit comes up - A. pay for more, B clear room. C Both  --- Consider- are data points from 2010 and 2011 needed anymore in ServiceNow even in archives- have all relevant reports been run regarding them been run years ago.  You could export in xml large traunches of old data and back them up somewhere safe or import them into a "Data Lake" or repository like PowerBI. SO my final resort would be xml exports to get data out.

Finally you can broach the topic with ServiceNow to get estimates on how much space does say 1 million records take? Do the updates ServiceNow pushes take up space otherwise allocated to you- does ServiceNow have a policy on increasing space for loyal customers that keep growing with ServiceNow and using the tool year over year. Things like that

[George P]

Archiving will not affect Performance Analytics (PA) reporting because PA stores the results of collections in another table.  The source data can change, but the previously collected PA data remains untouched.