mid server threads

[stevemac]

Hi,

We are running multiple MID Server agents on Windows 2012 VMs. 

• 2 VMs (each with 4 vCPU, 24GB RAM) each running 9 agents (Istanbul Patch 11)
• 1 x SOAP outbound integration (~3000 small outbound payloads per day) - that uses 1 specific agent on one server
• File transfers via JDBC File Loader from MID server file system to instance (about 120K rows per night)  - that is spread over 8 agents on the other server
• no discovery or service mapping

We appear to have a memory leak on the servers. We are rebooting the servers every 12 weeks (approx).  We will be getting a crash dump & submitting to Microsoft for investigation.  This servers are used only for ServiceNow, but they also have a myriad of other apps / services running

I do not have any background with Java applications & my Windows troubleshooting is pretty basic, so looking for some input on what is normal when looking at thread counts for the Java processes on the Windows server and on the instance (MID Server Threads [ecc_agent_threads])

From the Windows resource monitor

From the instance

 

Should I expect the thread count to be so high, or should the count drop when the agents are not busy (& when these images were taken - the agents were not busy)?

Any investigation recommendations?

This issue has been occurring for some time.  A historical look at memory utilisation suggests the issue commenced almost the same time we upgraded from Geneva to Istanbul Patch 6 & did not change when upgrading to Istanbul patch 11.  We are just kicking of London testing so should have an indication if the behaviour has changed in the couple of weeks.

thanks,

Steve

[stevemac · Accepted solution]

I ended up logging this with HI & got a very fast and helpful response.  As a summary...

• MID Server agent is having an issue with thread management
• the MID Server agent is unable to connect to AMB (asynchronous message bus).  That failed connection is not getting managed correctly.  As a result the thread count increments by 1 every failed attempt
• Failure to connect to AMB is probably an issue with firewall / proxy on our side (will troubleshoot this later)
• Workaround is to disable AMB via a MID Server configuration parameter (property = mid.amb_disable, value = false)

We implemented the changes in non-prod last week (including a restart of the MID Server agent hosts).  Production was done this week

regards,

Steve

[stevemac]

and actual root cause was an incorrect proxy host value in the MID Server agent config.  the value contained "http://" as well has the host.  Once we removed "http://" and enabled AMB it worked correctly