Application Services: What's new in Rome

David Skowronek · ‎12-31-2021

In my previous community articles Application Services: How to use them? and Application Services: Tips and Tricks I described basic usage and functionality of Application Services, with some useful tips related mostly to Calculated Application Services [cmdb_ci_service_calculated].

In this article I will focus to the Rome release and what has been improved with regards to Calculated Application Services.

The most important changes are not directly visible but they have major improvement to the instance performance and data accuracy:

significant performance improvements related to Service Mapping Recomputation job, that - in the worst case scenario with pre-Rome releases - could lead to memory issues of application node(s)
synchronous population of the CI Association [svc_ci_assoc] table on Application Service change

Performance Improvements

ServiceNow needs to keep service maps up-to-date, and recompute them when needed. With regards to Calculated Application Service it means that when a new relationship is created and/or existing relationship is remove, service map has to be validated and re-build.

This operation is very heavy, especially for very large maps having higher 100's or 1000's of Configuration Items. Recomputation is done by the scheduled job "Service Mapping Recomputation" that is set to "ALL ACTIVE NODES" - there is instance of the job running on every generic or worker node (in case of UI and worker node separation). Moreover, there can be more than 1 job per node, depending on the "glide.service_mapping.recomputation.job_count" property.

With pre-Rome release this job consumed a lot of memory for large maps, and implementation recomputation logic was not optimal. As a result, for large maps, this job was running for a long time. Run time (recalculation of a single large map) could take 30+ minutes.

This job itself could not lead to memory issues, this could happen when it was combined with another heavy job(s) assigned to the same application node. As a result, application node could be extremely slow and users being logged to this node could experience significant performance issues. However users loggded to other nodes would not see any issues at all. That makes the root cause analysis more complicated, as it happens highly randomly, and only subset of users are impacted for limited period of time.

This behavior has been addressed in Rome release, where Service Mapping Recomputation job has been redesigned. As a result there are no high peaks in memory usage caused by this job, as memory operations done previously in memory were migrated directly to SQL queries. This led to both memory consumption improvements as well as significant reduction of the recomputation time.

Based on the tests provided by the DEV team, and confirmed in real environment of a customer I am engaged with (having > 10 000 Calculated Application Services), for large maps (having higher 100's or 1000's of Configuration Items) the calculation time required is reduced by more than 75% (in some cases more than 95%), with no memory issues after Rome upgrade.

Even with those significant performance improvements that allows to use very high amount of Application Services, keep in mind sizing recommendations I have described in Application Services: How to use them?, section "Summary".

Synchronous population of the CI Association

As I explained in the article Application Services: How to use them?, section "What is a CI Association", some features like impact analysis depend on the CI Associations [svc_ci_assoc] table. With the pre-Rome release, this table was populated by a scheduled job that could result in a significant delay, and impact analysis may not be accurate.

Starting with Rome release, population of the CI Association table is synchronous, ensuring that CI Associations are update to date and impact analysis for e.g. Change tickets is accurate. This feature is enabled by default for both new customers and upgrades, and you may verify by checking the system property "sa.svc_ci_assoc.synchronous_population_enabled". Value "true" means that synchronous population is enabled.

mbourla · ‎10-27-2022

Hi David,

Is there any information available on the logic used to regenerate or refresh the svc_ci_assoc table? We don't have Service Mapping but we have long had a custom table that fulfils the same purpose i.e to say which CIs are associated with which services due to being somewhere in the Service's dependency/relationship tree. We've worked to optimise our code but it still takes many hours to run the daily job. I'm really interested to see if there is information about how ServiceNow have made their processing as efficient as possible. I believe it's scheduled job "Service Mapping - sync svc_ci_assoc" that does the work, but as I say we don't have that, and I imagine the code is hidden anyway. Any information on how it works would be really useful.

Thanks
Michael

David Skowronek · ‎10-28-2022

Hi, @mbourla . You are right that the logic is done by the mentioned scheduled job. And it is all hardcoded. Trying to do it by custom script is possible but very, very bad from the performance perspective (I did that about 5 years ago). Now you have baseline solutions available that take care about the synchronization in a fully optimized way. Calculated Application Service is doing near-to-realtime sync between relationships and CI Associations. Please check my another article https://www.servicenow.com/community/common-service-data-model/application-services-how-to-use-them/... for more details.

mbourla · ‎10-31-2022

Thanks, @David Skowronek. Our situation is that we are still early in our journey to migrate over to CDSM and at present have close on 10,000 Service CIs. In time they will be reclassified to other classes but for now they are just plain Service CIs. For maybe 10 years we've had a custom 'flattened' table of Service to CI relationships, so that predates Service Mapping and the svc_ci_assoc table. That custom table is used for adding impacted services in Change and Incident, and also in other places across our instance, which is pretty customised. So yes as you say, as we move over to using things like Calculated Application Services, we can hopefully use svc_ci_assoc instead of our custom table. Until then, we have to keep using our custom table, and try to keep optimising the code as our CMDB grows in terms of number of CIs and relationships. That's why I was hoping to get some useful pointers from the OOTB code that maintains svc_ci_assoc, which I could make use of in our own custom scheduled job. However I can fully understand if it's not information that ServiceNow are prepared to disclose. But it was worth a try!

marshallpar · ‎02-10-2023

Where do we track when & how Dynamic CI Groups refresh the list of associations [svc_ci_assoc]?

I see the system property "sa.svc_ci_assoc.synchronous_population_enabled" is enabled, and no longer calls a scheduled job but the CI Associations are updated synchronously for Calculated Application Services.

However, we have many Dynamic CI Groups where the result of the CMDB Group query is substantially larger than the number of CI Associations for that Dynamic CI Group - and not sure where this is supposed to be calculated/refreshed dynamically.

Barry Kant · ‎02-11-2023

Hi Marshall,

isn't the scheduled job 'Update svc_ci_assoc and Error Counters' active and set to Run periodically with a certain Repeat Interval?
That interval is the answer.

Best regards,
Barry

marshallpar · ‎02-13-2023

The scheduled job 'Update svc_ci_assoc and Error Counters' is interacting with a System Property called 'sa.svc_ci_assoc.synchronous_population_enabled' that was set to true as of the Rome release, and is only updating Counters - no longer generating svc_ci_assoc updates on a periodic basis.

While this works well for the Calculated Application Services [as changes are done synchronously], I think this may have introduced a gap for calculation of Dynamic CI Groups. Trying to figure out how best to solve that gap.