Backup Storage increased after starting to use Calculated Application Service

[Henrik Jutterst]

We moved all of our Application Services from Application Services [cmdb_ci_service_auto] to Calculated Application Services [cmdb_ci_service_calculated] what we then started to see was that our storage started to increase day by day ad week by week.

The table, [em_impact_graph] went through the roof and keeps growing where we now have over 235 000 000 records in that table. This effects storage licens that we are using and I'm wondering if there's a way to limit the size or reduce the size of this table.

We are using Discovery to populate CMDB and started to use Calculated Application Services so that we could have an updated Service Map for our Application Services to be used in Event Management.

But I did not expect these volumes and we are now blowing up storage limit.

Question: What is a causing this - and is there a way to limit the table to keep growing?

Impact Graph table [em_impact_graph] contains roughly 235 000 000(!) records

Base Configuration Items table [cmdb] contains roughly 3 500 000 records

Anyone with some experience within the area here?

This table show it all, how it is increasing:

Digging into an Application Service and looking at a specific Child Name gives me this (and it feels odd/wrong) and I grouped it:

Reading up on this KB-article: Event Management - Impact calculation explained 

Kind regards

/Henrik

[Henrik Jutterst]

Still no clear solution found to what is causing this issue - however we seem to found something interesting.

In dev-instnace, we removed the "Manual Endpoint" CIs and after that I can see that records in Impact Graph went down by 99,996%.

Still waiting for feedback from ServiceNow Hi if/what/why this is happening.
To me this sounds like a bugg.

Fig. 01 - Table audit in Dev after removing Manual Endpoint.

[Henrik Jutterst · Accepted solution]

This seems to have been a know bugg in ServiceNow. We upgraded to "Utah - Patch 6" and issue was resolved by itself.

Background: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1437164 

Impact Graph [em_impact_graph] "deleted" records deletes only after 3 month instead of 2 hours

Description

• It is observed that there is a huge number of records present in the em_impact_graph table with the status as Deleted but the actual record is not getting deleted.
• The number of records is getting added regularly thus causing issues. This is seen after the introduction of a new deletion job "Event Management - Clean Impact Graph Table" in Utah version.

Steps to Reproduce

1. Create several Monitored services.
2. Make sure impact calc is triggered.
3. Retrigger impact calc (so that entries in em_impact_graph table will be in status deleted)
4. Check after two hours or more and see if deleted entries are kept for much more then two hours |two days|two weeks|two months.

Workaround

• In order to fix this issue, we can add the below system property and execute the job.
  • Name: evt_mgmt.impact_calculation.cleanup_age_seconds.em_impact_graph
  • Value: 7200
    (seconds = 2 hours)

Related Problem: PRB1676869