Maximum number of Indicator Tasks

[Max Mirian]

Introduction:

Many of our customers are using SN GRC/IRM applications and they are getting advantages more and more. One of the powerful components is Indicators, enabling the customer to automate evaluation of KPI/KRIs, and many other use cases like manual evaluation of controls etc...

In the platform, we have a property which determines and controls the number of Open Indicator Tasks. which is by default set to 1000.  

Many of our customers have noticed that 1000 is not enough and consider increasing the allowed number of open indicator tasks, and at the same time they are worried about the consequences of this change and any possible negative impact it could have.

This article is to cover the topic of changing this number, increasing it to 10K or 50K, and see what impacts it can have on the performance.

Analysis:

Taking a close look at the data model reveals that Indicator Tasks in the GRC scope are extending task table. The task table has its own logic and workflows built around it. This requires that tasks (including grc indicator tasks) to be kept in manageable levels to ensure performance and usability it not negatively impacted.

It is advised to query always active records. also archive closed tasks after a certain time like 1 year and trying to maintain the number of active records below 10% of overall number of records for optimal performance.

Since Indicator tasks are generated only for the indicators of type Manual. This kind of indicator do not search for evidence data, and a task is created instead. The automated indicators, will have their values based on the query or script and do not enforce any tasks to be created.

A scheduled job 'GRC Indicator nightly run' scans through all manual indicator tasks and considers the execution login. It first checks the value of 'Next run time'. If it matches the current date, it counts the number of open indicator tasks and compare it to the maximum open indicator tasks stored in the GRC application property sn_grc.max_open_indicator_tasks. if thee threshold is reached and the current number of open indicator tasks exceeds the threshold, the creation will be aborted, and since no history is recorded, so the creation cannot be resumed. The creation proceeds as long as the number is less than the threshold.

The question which many customers ask is:

What happens if we increase the number by 10 or 50 times, in other words setting the threshold to 10 or 50 thousand?

Potential Impacts:

The performance impact caused by increasing the number of open GRC indicator tasks to 10K, 50K or even more only occurs during the tasks records creation period, only when the nightly job is running.

This change does not have any impacts on any users or any other jobs or even the overall performance whilst the job is running. The only noticeable impact is that the job completion is delayed.

We also conducted a performance test to assess the impact to platform users and users opening the indicators and  indicators tasks when 'GRC indicator nightly run' job is running. Here are the results.

• Increasing the property value to 10.000 ( 10 time the baseline value) resulted in 13.4% increase in indicator tasks creation completion time. Ten thousand indicator tasks got created in 13.4% more time than it took to create one thousand.
• Increasing the property value to 50.000 ( 50 time the baseline value) resulted in 165% increase in indicator tasks creation completion time. Ten thousand indicator tasks got created in 165% more time than it took to create one thousand.
• In both cases there is no impact on the heap memory consumption of the application node. This means no performance impact to any users / jobs or the overall platform even while the job is running.

Best Practices:

1. The value of grc application property 'sn_grc.max_open_indicator_tasks' can be adjusted by administrators. It should be documented and kept in a manageable limit.
2. 'GRC Indicator nightly run' job , as configured in baseline, should run during off-working hours. you can make a note on completion time.
3. Make sure there are not no other jobs, reports or time consuming logic runs in parallel and during the execution of nightly job
4. Ensure the number of expected generated tasks do not exceed the new threshold. Discarded Indicator tasks will not tracked and creation will not be resumed after abortion.
5. To ensure the overall optimal performance, archive closed tasks according to company policy (e.g. 1 Year) and always query active records.