ServiceNow Logging

[sjmeissner]

Who hasn’t clicked on the System Log links in ServiceNow with the hope of quickly finding an error message from a failed process… Only to be greeted with a list of several million entries that has taken minutes to return. What if you had to find the same error over multiple weeks and it took minutes each time you ran a search?

Managing logging on the ServiceNow platform is often seen as a challenge. Many processes write logs, and not everything follows a method to make them useful later. With a little bit of strategy and thought in how you write and access the logs, at least it’s possible to reduce some of the frustration.

Depending on what role you are in, consider the following practices.

 

Developers

• Script Logs
  • Use the Script debugger wherever possible. It is always best to prevent logging when it is not required. (https://developer.servicenow.com/dev.do#!/learn/learning-plans/utah/new_to_servicenow/app_store_lear...)
  • If you must write to the system log, ensure you include well-structured information, make sure the source field is populated and that you are using the proper functions (gs.log is out, and gs.info, gs.warn, and gs.error are preferred)
  • If logging is required for your code, consider;
    • Where should you be writing the log entry? Is the system log really the right place?
    • How will you turn your logging on and off? This could be important if it’s only debug logging or is not required in some environments. Using a system property is the most likely start point here.
    • Ensure that your logging information is helpful. Writing the output of a function or a variable value on its own does not help. Ensure you include enough description to know which variable or function and what the logged output should represent.
• Flows
  • Avoid setting the global system property for Flow recording (com.snc.process_flow.reporting.level) to anything other than “off“. 
  • Enable flow recording for the flows that you are currently interested in using”Flow Settings”. ( https://docs.servicenow.com/bundle/washingtondc-build-workflows/page/administer/flow-designer/task/e... )
  • Tip – setting the system-wide Flow debugging property (com.snc.process_flow.reporting.level) can cause significant performance issues in production environments, and I’ve seen it cripple non-production environments.
• Other Modules – Discovery, CMDB, ITSM, etc…
  • Ensure you review the OOB system properties for new ServiceNow modules as you are implementing them. Ensure you have the logging properties set correctly for each environment you are deploying to.

 

Administrators

• Avoid using the default navigator links when accessing large logging tables (e.g., System Log > All / Warnings / Errors), as these often default to all entries for today. Use “syslog.filter” (or equivalent) and set a “created date” filter for the smallest window possible to avoid long load times.
• Tip – Consider altering the menu items or using a business rule to add a date filter if the user has not included one.  This can help avoid long wait times for results on busy instances.
• If you have found the entry you are looking for but cannot determine which piece of code generates it, use the context map to help identify it. Check out this article for how – https://www.servicenow.com/community/developer-articles/syslog-context-map-is-making-my-life-so-much...
• If you are already collecting logs into a central logging tool (e.g. Splunk, Elastic, Sentinel, etc.), consider opening this tool for troubleshooting and general log review.  This can be especially helpful in production environments and enables a lot of debugging without ever needing to login as an Admin.
• Identify other tables that should be treated the same way (System Log, Emails, Event Log, Discovery Logs etc.)
• Review your log-related system properties when onboarding new capabilities. The settings for the Non-Production development environments are likely not what you want to run in Production.

 

Platform Owners

• Ensure you have a documented Logging Strategy in place. The key areas that should be covered include;
  • Logging Levels: (Info, Warning, Error, Debug). Guidelines for when each level should be used should be established to make it easier to filter logging data later.
  • Standardisation and Formats: Consider adopting organisation-wide formats for your ServiceNow logging. This will help integrate your logging data and supplement the overall traceability across your organisation.
  • Data to Log: Consider what data can and cannot be logged. While this might seem obvious, you will need to ensure you are logging enough information to satisfy your requirements while avoiding logging any data that might be subject to privacy laws, etc.
  • Log Storage: Choosing where to store logs might be straightforward if your only option is on the platform. However, if you are integrating into a broader organisation environment, you might need to consider which target is best for what you are logging. e.g. Is everything sent to your central logging tool like Splunk? Do your Cyber / Security logs need to be sent elsewhere?
  • Retention Policies: Log retention is an important consideration, and there are often multiple aspects. Keeping log data for a minimum period is often the first point that comes to mind. However, make sure you check your legal and regulatory requirements. You may also have provisions that require data destruction after a certain period.
  • Security: Protecting your log data is often a critical step to be considered. You will likely need to consider who is able to access the different logs and you should always ensure your logs are tamper-proof, especially where they fall under regulatory or legal obligations.
  • Performance Impacts: Generating too much log information can place undue load on the platform which can lead to a general slow down. Understanding the impact may lead to implementing further strategies like switching logging on/off as required, asynchronous logging or even rate limiting.
  • Compliance and Auditing: Regularly reviewing both your logs and logging strategy will ensure that new code being introduced onto the platform is compliant and no issues are creeping in. It also important to consider the impact of Regulatory or Legal changes which may require changes to the overall strategy.
  • Exception Management: Data leakage through log information can be a tricker area. How would you handle someone’s personal information being written to a central logging platform that made the data more accessible? Can you remove it? What about your tamper-proof requirements? Establishing a clear process will ensure that your teams know what to do and also ensure that you do not breach any regulatory requirements.
  • Log Monitoring and Analysis: Log information is written for a reason. Not monitoring or leveraging that data often misses an opportunity to identify a problem before it becomes a real issue. Define a structured process for reviewing the logs regularly and determining when a shift or acute issue has occurred or is developing. When leveraging a central logging platform, consider leveraging the available alerting and anomaly detection processes to reduce the human eyes on glass component.

• It is important to have a set of Development Standards that identify how the logging strategy is to be applied when developing on the ServiceNow platform. I recommend providing practical examples so that all developers can ensure they align their code appropriately.
• Ensure you implement processes to review and sign off on logging configuration during deployment and acceptance into Production. This could include a simple scan to ensure no new “gs.log” statements are being introduced. It might also include a more comprehensive review of what information is being pushed into logs to ensure it can not involve PII etc.

Log management on the ServiceNow platform is often one of the most overlooked areas. Investing some time up front to clearly document your approach, along with correctly enabling your teams goes a long way to avoiding any challenges down the track.