PowerShell Scripts causing memory to be full and shutting down the MID server

[teja2090]

We do have user provisioning done using PowerShell to add user to AD groups, Azure Groups and DLs before getting hired. The issue we are facing is the MID memory is getting full due to some memory leaks. What we identified is under the orchestrator user > AppData > Local > Temp folder there are 24MB folders being created continuously related to ExchangeOnline or Exchange. This is filling up the memory. Also, the PowerShell activities are getting stuck and blocking the RAM.

1. We have added a task in the task scheduler in the MID as well to delete the files in Temp folder every 30min.

2. Increased the RAM to 32GB

3. Limited the number of threads to 5 so that only limited activities run 

4. We set the heap memory to 4096 

We wanted to end this issue and do RCA. Can anyone please suggest a way to identify which PowerShell scripts are causing this and how to troubleshoot this further? We are no PowerShell experts so not able to exactly identify the lines in the code. There are nearly 90 scripts in our company.

[suvro]

Get in touch with windows admin team and get this sorted.

Below link can help

https://answers.microsoft.com/en-us/msoffice/forum/all/temp-files-from-office-365-2016/426adb42-e866-4249-b5f5-622ff57ffc74

[Ram Devanathan1]

Have you reviewed the powershell code?

Possible that some part of the powershell code is actually getting large list of user details or groups and that's causing the huge memory usage..

[Richard Hine]

Teja2090,

Have you considered that your directory cleanup script might be adding to the problem? Any task that is terminated by you cleaning up the folder might be re-initiated/retried and recreate the folder, so you could be perpetuating the problem.

You do not mention how many CPUs your MID Server has, but I think you could potentially be choking the machine by imposing the thread limit, SN recommend a quad core CPU with 25 concurrent threads. Might be worth having a look here : https://docs.servicenow.com/bundle/sandiego-servicenow-platform/page/product/mid-server/reference/r_...

Given the same situation, I think I would look to see if the orchestration workflow could be broken down, potentially moving some pieces to another MID Server, or seeing if some of it could be run async.

I encountered an issue with a very high volume Client Software Distribution environment interacting with SCCM via OOTB Orchestration. The powershell script files checked they could access the target system for every connection before initiating the connection which added to the resource burden on SCCM and it started to refuse connections.

In the end I worked with the SCCM administrators to rewrite all the powershell scripts as WMI calls (Powershell ultimately wrapped WMI in this case!) and then built new Activities to use the new scripts and completely reworked the OOTB orchestrations.

I am not saying this is your problem, but in high volume situations you sometimes have to take an alternative view of what is provided OOTB to ensure it meets your needs.

Hope this helps,

Richard