Great Catastrophy! - Restore CI using "Restore Record and References"

[oliolio]

Hi,

I've got a couple of questions

1. So one of our CMDB admins accidently deleted a CI that has been used in a lot of records historically (around 200 000 records according to our estimates) and we went ahead and restored the CI using the "undelete" function without knowing that it would not be restoring the references. Currently we have a very large number  of uncategorised incidents, requests and changes. Is there any way to recover from this? 

Unfortunately this is not the only CI that was deleted. The CMDB admin also managed to delete a number of CI's used in roughly ~10 000 tickets.

2. After activating the Recover deleted Records plugin we went ahead and tested it out in our test environments. However, it does not recover even 25% of the references. We checked our audit tables and it seems that in some (not all) the CI is removed from the Sys_audit table after deletion. That is we can see it in the history before we delete it but when we check the ticket after deleting the CI it looks like if the ticket never had a CI to begin with. Anyone seen this behaviour and got a solution? 

We are still on Jakarta but in process of upgrading to London. How does the "Recover entire operation" work? Anyone here got any experience with that one? 

Anyone got any suggestions at all? We are a fairly large organisation with around 3000 individual tickets getting created each day. Going back even an hour is impossible for us so we cannot roll back in any way.  

[sachin_namjoshi]

This is scary situation.

Do you have custom code on your instance which removes records from sys_audit table?

I suggest to debug for any code ( business rules, script includes...) which is removing any records from sys_audit table.

Regards,

Sachin

[oliolio · Accepted solution]

"2. After activating the Recover deleted Records plugin we went ahead and tested it out in our test environments. However, it does not recover even 25% of the references. We checked our audit tables and it seems that in some (not all) the CI is removed from the Sys_audit table after deletion. That is we can see it in the history before we delete it but when we check the ticket after deleting the CI it looks like if the ticket never had a CI to begin with. Anyone seen this behaviour and got a solution? "

So it seems we've found the cause here. If there's been no change to the CI after the record was inserted the information will be removed from the audit and history line when the CI is removed. Hence servicenow is not able to recover it. 

Will post an incident on HI regarding this,