AWS Discovery Running Too Slow

[utkarshbk]

We're running AWS Cloud Discovery. Due to the large volume of assets, discovery is extremely slow. A complete run has ~3 million probes and takes about 3 days. We're running using 7 MID Servers.

I've been monitoring CPU, RAM & Network bandwidth usage, and they are all underutilized. Using more powerful machines doesn't work either, MID Servers aren't able to utilized the additional resources.

This leads to conclude that MID Servers not scale vertically. We have tweaked many parameters like "threads.max", "glide.mid.fast.responses" & "glide.mid.max.sender.queue.size" but we haven't seen any change in speed of AWS discovery.

So far, horizontal scaling has worked and made discovery faster, but I am wondering if using 10s of MID Servers is normal?

Also, can you provide recommendations to speed up AWS discovery?

[RawelS]

Hi Utkarsh,

Below mentioned are the steps that we performed in order to improve Cloud discovery performance.

1. Using MID cluster of 4 MID servers for cloud discovery

2. Increased CPU and RAM on all the 4 MID server hosts

3. Increased the JVM for each MID server (This is very important for performance and only increasing threads will not help)

4. Increased the threads on MID servers appropriately.

5. Ran the discovery in sequence using "Run -> After Discovery" option. This is very critical when running multiple schedules using same set of MID servers, as the performance gets hit if more than 1 schedule using same cluster are running in parallel.

6. Disable patterns/pattern steps for certain CI classes which are not at all required. We have filtered out millions of CIs from cloud discovery as there was no need for them. This also has saved us time/resources.

Doing this we are able to finish cloud discovery within 18 hours for all service providers (i.e. GCP, AWS and Azure)

Hope this helps you.

Thanks,

Rawel

[utkarshbk]

Hello @RawelS ,

Thank you very much for sharing your experience and for all of your suggestions. Particularly we worked on 3rd point (increasing JVM) because we'd already tried all of the other suggestions. Increasing JVM didn't work for us.

[utkarshbk]

Hello @RawelS ,

Thank you very much for sharing your experience and for all of your suggestions. Particularly we worked on 3rd point (increasing JVM) because we'd already tried all of the other suggestions. Increasing JVM didn't work for us.

[RawelS]

Hi @utkarshbk , Have you tried exploring Service Graph Connector for AWS ? It is recommended over cloud discovery and is efficient as well in terms of performance.