Great Catastrophy! - Restore CI using "Restore Record and References"

[oliolio]

Hi,

I've got a couple of questions

1. So one of our CMDB admins accidently deleted a CI that has been used in a lot of records historically (around 200 000 records according to our estimates) and we went ahead and restored the CI using the "undelete" function without knowing that it would not be restoring the references. Currently we have a very large number  of uncategorised incidents, requests and changes. Is there any way to recover from this? 

Unfortunately this is not the only CI that was deleted. The CMDB admin also managed to delete a number of CI's used in roughly ~10 000 tickets.

2. After activating the Recover deleted Records plugin we went ahead and tested it out in our test environments. However, it does not recover even 25% of the references. We checked our audit tables and it seems that in some (not all) the CI is removed from the Sys_audit table after deletion. That is we can see it in the history before we delete it but when we check the ticket after deleting the CI it looks like if the ticket never had a CI to begin with. Anyone seen this behaviour and got a solution? 

We are still on Jakarta but in process of upgrading to London. How does the "Recover entire operation" work? Anyone here got any experience with that one? 

Anyone got any suggestions at all? We are a fairly large organisation with around 3000 individual tickets getting created each day. Going back even an hour is impossible for us so we cannot roll back in any way.  

[oliolio]

Unfortunately we are running an on premise installation so we cannot contact Servicenow Support in these matters as they do not have access to our servers. It seems we will have to go for database backup (i'm hoping we have a snapshot we can use that is current enough otherwise we will need to find another way) but we really need to find out why the built-in restore function is not working as intended and why the audit is getting deleted.

We would only be able to use the backup if it only have an effect on the CI's and tickets affected. If not then DB backup is out of the question.

[sachin_namjoshi]

Restore with reference works with below exceptions

• Record deletions are not tracked on tables with the no_audit_delete=true dictionary attribute.
• File attachments are not restored when the record is restored.
• References to a restored record are restored only if the reference field is on an audited table.
• References to images using an Image type field are not restored.

You can run below one time fix script to restore CI and their references ( e.g on incidents, change request...)

function restoreReferences(tableName, fieldName){

var brokenRecord = new GlideRecord(tableName);

brokenRecord.addNullQuery(fieldName);   // No use processing records where there is still an AG

brokenRecord.query();

while(brokenRecord.next()){

        var lastFieldValue = new GlideRecord('sys_audit');

        lastFieldValue.addQuery('documentkey', brokenRecord.sys_id);

        lastFieldValue.addQuery('fieldname', fieldName);

        lastFieldValue.orderByDesc('sys_created_on');

        lastFieldValue.query();

        if(lastFieldValue.next()){

                  brokenRecord.setValue(fieldName, lastFieldValue.newvalue);

                  brokenRecord.update();

        }      

}

}

Regards,

Sachin

[Yifeng Zhang]

Well if he hasn't modified the old 200000 incident records, then the old reference id is still there, it will just link to nothing, Just loop through the ticket table, check for those faulty reference and replace it with new ones...... since it's only 10 ish CIs , just hard code the reference in there... 

[oliolio]

Ah, now i see what you are saying.

However, unfortunately the ID is not there. It's empty with no reference. As i wrote earlier, for most of the tickets even history of ever having data in the CI field was removed when the CI got deleted. We have been able to reproduce this in our test environment but we have no idea why it happens.

[Yifeng Zhang]

Oh ... in that case the association information is lost.. 😞 that would be tricky, what I think would work is if you could restore a second copy to the time prior to deletion and find that association information... but yeah, it would require a restoration to another instance.