Yesterday I was on a call with a customer who had a tough problem: Discovery was working fine except for network gear, and even with heroic efforts they'd been unable to find the problem. They'd identified the bit that was failing: the PortScanner probe was reporting that the network gear was not responding to SNMP, and therefore the SNMP probes were never getting fired.
Now SNMP — as its name implies — is a pretty darned simple protocol. There's not much to go wrong beyond credentials (the community string) and some kind of access restriction. The customer had already eliminated access restriction as a problem: they tried running another SNMP tool from the same computer the MID server was running on, and that tool worked just fine. That left credentials as a source of trouble.
So we looked at the credentials (Discovery → Credentials) and saw something like the screenshot at right. Looks fine: two SNMP credentials, marked "active". And the customer had re-entered the credentials several times, very carefully, to eliminate any possibility of fat-fingering. The credentials looked fine — so what the heck was going on here?
At this point we broke out the big gun: we installed WireShark (an excellent free open source network sniffer) on the MID server, and watched what was actually happening on the network as Discovery tried to explore a piece of network gear. We also watched the other SNMP tool (the one that worked!) to see what it was doing differently.
Well, the other SNMP tool was behaving differently all right: it was using the correct community string and Discovery was trying only "public" (the default community string that it always tries). Somehow Discovery was managing to ignore the credentials so carefully entered. But why?
At this point we went back to the Credentials table and drilled into one of them (see screenshot at right) — and then the problem was obvious! For reasons unknown, someone had configured the credentials to apply to specific MID servers, instead of the default of all MID servers. No problem so far, but…then they neglected to specify any MID servers! So the credential was there, it was active, but it applied to exactly zero MID servers.
It was a good credential gone bad…
We set it All MID Servers and ran Discovery again, and everything worked just fine.
