Resolving MID Server issues

  • Release version: Washingtondc
  • Updated February 1, 2024
  • 3 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Resolving MID Server Issues

    This guide provides essential troubleshooting steps for resolving MID Server issues, monitoring their performance, and understanding Java Flight Recorder (JFR) functionalities. It aims to equip ServiceNow customers with the necessary tools to ensure their MID Servers operate effectively.

    Show full answer Show less

    Key Features

    • Monitoring the MID Server: Utilize log files, network checks, and MID Server status verification. Ensure the MID Server process is active and monitor its memory usage.
    • Java Flight Recorder: JFR allows for detailed analysis of JVM memory and thread usage, with options to start, stop, and write recordings. Configurable parameters can control data retention and event writing.
    • Troubleshooting Procedures: Specific procedures and Knowledge Base articles are available for various MID Server problems, including connectivity and resource issues.
    • Active MID Server Issues: The system tracks active issues with MID Servers and provides alerts for resource over-utilization, credential issues post-cloning, and user connectivity problems.
    • User Debugging Output: Enables summary and detailed views of user issues, which must be activated manually.
    • Upgrade History: Review upgrade processes and errors through the MID Server Upgrade Histories and Stages tables.

    Key Outcomes

    By following these guidelines, ServiceNow customers can effectively monitor and troubleshoot their MID Servers, ensuring optimal performance and timely resolution of issues. Implementing alerts and utilizing JFR can further enhance system reliability and operational efficiency.

    Troubleshoot problems with the MID Server to find solutions. Monitor the MID Server to receive alerts about issues as they occur. Troubleshooting procedures exist to resolve specific problems with the MID Server. The Knowledge Base on Hi contains several articles to help you troubleshoot MID Server issues.

    Monitor the MID Server

    Monitoring a MID Server involves verifying entries in log files, confirming network connectivity, and checking MID server status.

    • If the MID Server is on a Windows host, navigate to the Windows Services console, locate the service name that matches the name that appears from the wrapper-override.conf file. If the MID Server process is the only Java process running on the host, monitor the memory used by java.exe and alert on less than the maximum configured memory defined in the ~\agent\conf\wrapper-override.conf folder.

    • Ensure that the agent0.log.0.lck file appears in the ~\agent\logs folder to confirm that the MID Server running and logging system activity in the agent0.log.0 file.

    • Review the following logs for warning, critical, and severe errors: ~\agent\logs\agent0.log.0 ~\agent\logs\wrapper.txt See Manage ECC Queue content for a MID Server to see how to open log entries from the instance.

    • Test MID Server connectivity to confirm that the MID Server host is able to communicate with the instance. Resolve any issues with the network connectivity such as conflicts with firewalls, access control lists, and routing errors.

    • From the MID Server instance, navigate to the MID Servers page, and review the status of the MID Server. For additional information, click a Name.

    • Set up email, SMS, and push notifications to alert you when issues occur with MID servers. The MID Server Down notification is enabled by default. See Notifications for details.

    Java Flight Recorder

    Java Flight Recorder (JFR) uses an articulate data model to analyze Java Virtual Machine (JVM) internal and application specific memory usage and thread related details. On hosts with Java 14 or later, Event Streaming enables analysis of specific JVM or application events.

    The JFR file actions Start JFR Recording, Stop JFR Recording, or Write JFR File, are accessed on the instance in the MID Server's Related Links. By starting and stopping the JFR recording, you can dump JFR data to the disk on demand. While JFR recording is on, you can write to the JFR file from the MID Server instance page.

    The JFR file is deleted by a scheduler to clean the data after 24 hours. This default behavior can be overridden by adding the following configuration parameter in the config.xml: <parameter name="mid.jfr.data.clean.interval" value="<time in milisec>"/>

    Custom JFR events can be written to JFR file. To disable event writing, add the following configuration parameter to the config.xml: <parameter name="mid.jfr.record.aprobe" value="false”/>

    Troubleshooting procedures

    The following procedures provide information on resolving specific problems with the MID Server. Knowledge Base articles also cover a wide range of issues. If you are encountering problems which are not covered in these procedures, search the Knowledge Base for related articles. For instance, if you are experiencing symptoms such as the MID Server going down or not responding, or CIs being duplicated during discovery, see KB0597571 for information and recommended solutions.

    MID Server active issues
    The ServiceNow® instance has a dedicated table that publishes active issues with MID Servers and alerts administrators when a MID Server is in danger of exceeding its resources.
    Active MID Server post-cloning credential issues
    The system provides automatic processes to detect and notify you of possible MID Server credential issues after instance cloning.
    MID Server resource threshold alerts
    The instance displays warnings when a MID Server breaches its resource thresholds for CPU and JVM memory usage, enabling users to create email notifications or custom scripts when a breach occurs.
    MID Server user connectivity issues
    The instance writes issues involving MID Server user logins and network connectivity to the MID Server Issue [ecc_agent_issue] table and creates events you can use in custom scripts or to send email notifications.
    Interpreting MID Server user debugging output
    Debugging output from the system log is available in either a summary or detailed view for MID Server user issues, but must be enabled manually.
    MID Server Upgrade History
    Use this module to troubleshoot errors that occur during the MID Server upgrade process. The MID Server Upgrade Histories table contains a record of each instance upgrade. The MID Server Upgrade Stages table shows the status of each MID Server and its upgrade progress, including any errors encountered.