Great Catastrophy! - Restore CI using "Restore Record and References"

[oliolio]

Hi,

I've got a couple of questions

1. So one of our CMDB admins accidently deleted a CI that has been used in a lot of records historically (around 200 000 records according to our estimates) and we went ahead and restored the CI using the "undelete" function without knowing that it would not be restoring the references. Currently we have a very large number  of uncategorised incidents, requests and changes. Is there any way to recover from this? 

Unfortunately this is not the only CI that was deleted. The CMDB admin also managed to delete a number of CI's used in roughly ~10 000 tickets.

2. After activating the Recover deleted Records plugin we went ahead and tested it out in our test environments. However, it does not recover even 25% of the references. We checked our audit tables and it seems that in some (not all) the CI is removed from the Sys_audit table after deletion. That is we can see it in the history before we delete it but when we check the ticket after deleting the CI it looks like if the ticket never had a CI to begin with. Anyone seen this behaviour and got a solution? 

We are still on Jakarta but in process of upgrading to London. How does the "Recover entire operation" work? Anyone here got any experience with that one? 

Anyone got any suggestions at all? We are a fairly large organisation with around 3000 individual tickets getting created each day. Going back even an hour is impossible for us so we cannot roll back in any way.  

[Yifeng Zhang]

Given the description.... I'd start coding, write a script to manually match the old reference sys_ids to the recovered CI item... This might be your best bet with your situation.  Since it's historical records (no changes coming in), it's finite (only couple of CI is deleted), it's one time only... Spend an afternoon and fix it might be the way to go.

Just my opinion.

[oliolio]

We would do that if we were able to find out which records we need to update with the the recovered CI item. But this would require us to manually read through the description of each individual ticket to find out which of the ~10 CIs to match it with. And continue to do so with around 200 000 tickets. 

I'd say that this would take more than an afternoon. Or even more than twenty afternoons.

[Yifeng Zhang]

Well,

so let's say 10 CI were deleted and recreated.

So there will be 10 distinct sys_ids stored in those 200000 records that are broken.

loop through those 200000 records put them into 10 lists, so you know which one links to which,

do a simple update to the link them to the new CI.

I'd say it's half an hour job.

Here's a more detailed approach:

let's say 

This CI  {Computer Broke} was deleted and recreated with new sys_id : "abc123";

take a look at a couple incidents record that you would recognize that was suppose to reference this CI. Then let's say you notice they all commonly reference an old id : "bbbbbb";

Now you are fairly certain "bbbbbb" is the old ID for CI {Computer Broke}.

so your script, loop through all your 200000 records, if you see "bbbbbb" replace it with "abc123";

Tada  your link is restored.

If you only have 10 ish CI deleted.. this shouldn't take long.

[sachin_namjoshi]

We also faced similar tragedy in my old organization and we ended up creating P1 HI incident for service now support for them to apply latest database backup on your instance.

Service now support will take latest DB backup for your instance and then restore your instance with previous backup.

Also, service now support should help you with fix script to restore all CI references which you verify in non prod instance first before moving to PROD instance.

Regards,

Sachin