Different outcomes when Scheduling Discovery vs running schedule On Demand

[Steve Phayre]

Hi all,

Utah instance (although only recently updated and was seeing similar behavior in Tokyo) doing IP based server discovery.

Recently, we setup a discovery schedule using new MID servers to scan a newly deployed environment (on-prem, mostly VMWare VMs & Cisco physical hardware).  The schedule was set to run daily, and was scanning about 600 devices, and about 70% of the servers were being discovered as expected.  The remaining 30% was successfully getting past Shazzam with open ports properly identified, but was failing on Classify (no credential was successful even though all 4 Windows creds available to the MID server were attempted and failed).

On doing some troubleshooting of the servers that failed, I attempted to scan each one using the "Discover Now" action, and for each of the once that failed every day on the scheduled version, they scanned completely successfully!  So then I went into the Discovery Schedule, and ran the schedule from the UI action and if completed the scan, and successfully picked up the servers that had failed Classify on all of the scheduled attempts.

Has anyone seen this type of thing before?  I'm pretty sure I had this happen years and years ago, but I'm not recalling what solved it last time.

I'm relatively sure that the recent upgrade isn't to blame, because the upgrade happened in the last few days and there were failed scheduled attempts both before and after the upgrade.  And these servers (and IP addresses) had never been successfully discovered before, so there is no credential affinity records or anything like that.  I thought it might be something to do with the behavior that is selected when you click on Discovery Now on the CI, but as I understand it, if you do "Discover Now" on the Discovery Schedule, it will use the behavior/functionality selected in the schedule in the same way that it runs when it's executing on the schedule.

Has anyone else seen failures in scheduled discoveries that work fine if you kick them off manually?

[orons]

If you are saying that the schedule contains a single IP, then as a high level it is probably better to check how the ECC queues look like

The only main differences I can think of from the top of my head are that and MID server picked for the job [and therefore the network route being used to get to the device]

At least based on the high level description of the problem.....

But the best way is to record traffic on the MID : ) That way you will know for sure which party doesn't do its job

[Rinkus]

Hi Steve,

Have you noticed how your sys_trigger table ready state queue sensors behave during this failures?

[pratiksha5]

Try this 

When schedule discovery runs it sends a lot of requests at a time.  We have done the following with this discovery and were able to pick up devices correctly. We have monitored the traffic and understood that when schedule is running the target devices were not responding. 

- source: mid-server IP
- destination: destination ip range
- ports: allow all
- IPS: turned off all intrusion prevention checks
- App Control turned off
- limits: allow to exceed global session limits (this was causing most of the issue)