Prevent performance issues caused by improper use of system properties

[catie_carmody]

System properties are a type of global variable that are either stored in the System Property [sys_properties] table or set within the code base. They typically store configuration settings such as the format for a date/time field. Some properties are available on a system properties form, but some lesser-used properties are only available from the System Property table. In some cases, the property does not exist in a base instance, but can be added if you need to change the value. See the Available System Properties product documentation for more information.

It's important to use system properties appropriately, otherwise it may lead performance issues and even outages.

When should you use a system property?

System properties should be used to store configuration information that rarely or never changes. For example:

• Property: glide.ui.activity.style.comments, Value: background-color: WhiteSmoke
• Property: glide.ui.can_search, Value: itil,text_search_admin,admin

What can be potential performance side effects from the use of system properties?

Each time you change or add a system property, the system flushes the cache to keep all nodes in the cluster in sync. This cache flush has a very high performance cost for 1-10 minutes, which can potentially cause an outage if done excessively. To prevent such outages, do not use a system property to store configuration information that changes more than once or twice a month. Instead, use a custom table to store regularly changing configuration information.

What is a cache flush?

When a cache flush is running, you can see the following in the localhost logs:

2016-08-24 13:08:01 (197) worker.7 worker.7 WARNING *** WARNING *** Starting cache flush
2016-08-24 13:08:01 (197) worker.7 worker.7 Flushing cache
2016-08-24 13:08:03 (620) worker.7 worker.7 Loading properties from DB
2016-08-24 13:08:03 (671) worker.7 worker.7 WARNING *** WARNING *** Cache flush complete
2016-08-24 13:08:03 (672) worker.7 worker.7 Loading properties from DB
2016-08-24 13:08:03 (735) worker.7 worker.7 Slow business rule 'Properties change' on sys_properties:<system_property_name>, time was: 0:00:02.53

Some important things to point out:

• Flushing the whole cache (/cache.do OR changing system properties) is different from flushing individual caches. Flushing the entire cache occurs when we rebuild the entire cache, leading to potential performance degradation during the rebuild.
• All cache flushes propagate to all nodes (can see all cache flushes in the sys_cache_flush table).
• While one cache flush will not result in instance-wide performance degradation, doing flushes many times in a row can cause performance issues.
• When a cache is flushed, all queries go to the database until the cache is rebuilt, which may result in: higher response times, high CPU on application nodes, and high CPU on database servers.

See KB0610620 for example of an improper use of a system property by the MOOG Integration.