The primary system log transaction table in a ServiceNow instance is the syslog transaction table. This table logs information in regard to imports, workflows, configurations, errors and warnings as well as upgrades and other configuration changes within the instance. Hence, they have a tendency to grow and cause performance issues. There are some best practice scenarios you can follow which will help you avoid exponential growth of syslog tables.
I'll show you how to:
- Organize transaction tables
- Identify when there is excessive logging occurring
- Find the cause
- Prevent further performance issues from excessive logging
Organization of the transaction tables.
Both the syslog and the syslog_transaction tables (client transactions logs) are under table rotation. What that means is that they have a base database table, which has been sharded. Each table shard is defined by the number of days it will keep data and once that constraint has been met the inserts will move on to the next shard.
Table rotation syslog example
Here is a snapshot of how the table structure is of a table under table rotation. The syslog(Base table) has 8 rotations. Hence syslog0000 to syslog0007 table shards. The data is rotated between the shards and is overwritten every week. Only a month and a half worth of transaction data can exist in these shards.
The simple advantage of having a table under table rotation is that there is no single large table which keeps growing and data gets rotated into different sharded tables based on duration. This is an auto cleanup functionality. Once the series is over and the first shard is overwritten again, then the cycle continues. In this scenario we can never have more than a week's worth of data in a shard. By default, ServiceNow has taken into consideration many factors and have come up with a standard configuration which is what is seen deployed on a ServiceNow instance. Care should be taken not to modify these values.
How to find the source of excessive logging.
- Navigate to System Logs > All
- Select a source to group by.
You can modify the filters as per your preference.(Keep the row count to 20) and filter smaller results sets using sys_created_on for today and increase the time accordingly.
In this example, the scripts were causing excessive loggings. Drill down further and pull up the system logs to see the log statements and find the offending script include/job from where the gs.log() statements are coming from.
Getting the above result can take time if the shard is big(excessive of 1m records)
What causes excessive logging and how to prevent it
The two major issues that can happen due to excessive loggings are tables growing excessively large and CPU load increasing for the application servers, thereby causing overall performance issues.
- Excessive use of gs.log() statements, especially in scripts executing scheduled jobs etc with recursions. Try to minimize these and only use it where necessary.
- Scripts with errors. Check the error logs for details and try to rectify them to reduce the number of errors in them.
- Using debug properties for troubleshooting issues. Although this may not put inserts into the syslog table but they can cause a high CPU load due to being called with every transaction. One such property is the "glide.rest.debug" property. Once troubleshooting has been completed set them to false.
Following these steps can ensure better performance and response time for the instance. The more we adhere to this the less maintenance changes are performed by customer support on these instances. Like sergiu.panaite said in Performance considerations when using GlideRecord, "Don't overkill your script with logging statements!" Also, if your emails are opening very slowly from the list view of the sys_email table, you could try working around it by removing the email log embedded on the sys_email form.
3 Comments
