Custom SNMP Discovery Fails on Yokohama - All Handler/Probe Methods Exhausted

[Preston_83]

I've run into a very stubborn issue trying to discover a specific set of older Axis IP cameras on my Yokohama instance, and I'm hoping someone in the community might have seen this behavior before.

The Goal: Discover ~2,000 Axis IP cameras. A lot of these devices are older and do not have a standard serial number OID. My identification strategy is to get the device's MAC address and use that to derive a unique serial number and name.

The Challenge:

1. SNMP walk works: I have confirmed that running a simple snmpwalk.exe -v2c command from my Windows MID Server successfully retrieves the full MAC address table from the cameras. The data is accessible.
2. Patterns fail: Standard Discovery patterns fail. The SNMP Query operation in the Pattern Designer appears to use a Java SNMP library that has issues with SNMP bulk-gets on these older device firmwares, causing the pattern to fail and not return any data.

Because patterns were inconsistent with a chunk of devices on my network, I moved on to custom probes and sensors, but every attempt has been blocked by what appears to be platform-level configuration issues.

Core Configuration: I have already configured the foundational pieces for this discovery, and the initial steps are working as expected:

• SNMP OID Classification: I have a rule that correctly matches the camera's sysObjectID (1.3.6.1.4.1.368.1.1) and assigns the IP Camera class. The discovery logs confirm this classification is successful.
• IRE/Identifier: We have created a dedicated Identifier for the cmdb_ci_ip_camera class with multiple high-priority entries (Serial Number, Name, and MAC Address) to ensure that if any of this data were present in the final payload, the IRE would successfully create the CI.

Architectures Attempted:

1. Custom Probe w/ MID Server Script Include: I created a custom probe using the JavascriptProbe topic that called a MID Server Script Include. Result: This failed with the error MID Server Script Include output does not exist. I believe that the script file was never synchronized to the mid servers, even after correcting ACLs, scopes, and clearing the cache and restarting the mid servers.
2. Custom Probe w/ Embedded Script: I then embedded the script directly into the probe record itself to bypass the file sync issue. Result: This also failed with the same MID Server Script Include output does not exist error, which seems impossible.
3. Custom Javascript Handler (The Final Approach): I configured the out-of-the-box SNMP - Classify probe to collect the MAC address table and added a custom Javascript Handler (AxisHandler) to its javascript_handlers parameter, which I had to modify a field parameter because the field was limited to 40 characters.
• Result: The ECC Queue logs show the probe runs and collects the MAC address data, but my AxisHandler script is never executed. I confirmed this by placing simple gs.log() statements as the only code in the handler, and no logs are ever generated.
  
  

My Question for the Community:

Has anyone ever encountered a situation on the Yokohama release where the SNMP - Classify sensor seems to ignore the javascript_handlers parameter? Is there a known issue or an alternative architectural pattern for executing a simple command-line tool (like snmpwalk) and parsing its output that I haven't tried?

I have exhausted all configuration paths I know about and now believe the systems out of box snmp discovery probes are just not allowing my probes to run. I assumed that once the OID was classified it would trigger the probe I configured to trigger in the classification setup, but it isn’t triggering my probe. Any insight would be greatly appreciated.

Thank you!

[Fabian Kunzke]

Hey,

This sounds super interesting and i would like to see why patterns don't work. But to help out first:

Custom handlers on the classify probes - at least to my knowledge - usually don't work because the ootb handler just straight up continues with the identification phase without letting other handlers run. So it could be, that you additional handler is just skipped.

BUT: if that approach worked in your overall discovery, you may be in luck. All parameters returned by the classification probe can be pushed into the ci_data object (which then is available in later stages, even patterns in the identification phase). To best verify this, create a copy of the ootb sensor & have it triggered by the classification probe. Within the script of the sensor you should be able to assign the SNMP variable value to the ci_data object value.

Now, if both work, then you should be able to just create a identification probe & sensor for your camera. This can be as straight forward as just one SNMP walk being discovered with a sensor mapping just that one parameter.

I didn't see if you've already tried this (custom probe & custom sensor), but that would be what i'd try. No guarantee it will work, but it's worth a try. (Note: You can also skip the whole "classification probe" thing and go with an identification probe directly. Just add it to the triggered probes from you camera classifier).

Sidenote: If the MAC is indeed discovered, you don't need to derive a serial number from it. The MAC should be unique to a camera in combination with something like the host name.

Hope this helps (and if it doesn't, i will make sure to respond with other potential solutions).

Regards

Fabian

ps.: This is not at all uncommon and sometimes requires a lot of creativity (learned that through ~5 years of IoT discovery)

[Preston_83]

Thanks Fabian,

I have tried creating a custom probe/sensor combo with classification.  This is what I am running into. The payload shows the OID .1.3.6.1.4.1.368.1.1 with that leading period. My classification I created is OID Starts with 1.3.6.1.4.1.368.  I have made sure there is no other SNMP OID conflict starting with this OID, this is the only OID classification on my system that would return.  That classification then is configured to trigger my custom probe, but that isn't happening. My SNMP payload is showing 

name="probe_name" value="SNMP - Classify" not my custom probe.

I would love to get my pattern working, since it works with about half of my cameras.  This is what I am running into with my pattern. I am able to run snmpwalk against OID 1.3.6.1.2.1.2.2.1.6 from my mid servers and I get the Mac address from all of the devices I have tried.  That OID is a table lookup for ifPhysAddress. On the devices that don't work, this is my pattern error:
Finished step: 1 'Get Mac Address Table' with Error: Failed to get SNMP table; Empty/Null response

My snmpwalk -v2c -c "<password>" 10.36.86.14 1.3.6.1.2.1.2.2.1.6
IF-MIB::ifPhysAddress.1 = STRING: 0:0:0:0:0:0
IF-MIB::ifPhysAddress.2 = STRING: ac:cc:8e:4f:d3:a
IF-MIB::ifPhysAddress.3 = STRING: 0:0:0:0:0:0

So it works from a mid server snmpwalk from the command line, but my pattern can't find it, my theory is that the Mac address that is returned on this device isn't a standard octet format, the last octet is only one digit, or, it is a bulk payload, multiple lines and maybe the camera device doesn't support that?

If you have any ideas how to work around that with a pattern step I would love to hear it. We are on the Yokohama release and the pattern designer operations have changed a bit since the last time I wrote a custom pattern and I missing my script step.

Let me know if screenshots or any other details could help, I really appreciate it.

[Fabian Kunzke]

Oh, perfect. Then lets see if we can fix the pattern first, before trying everything else.

First point: You are getting a null response. If this only happens during the pattern execution, this might be related to timeout issues when retrieving the value. I'd suggest to follow this KB first: SNMP discovery may fail in classifaction and failing to get the sysObjectID or some OID related info...

Other than that, you can try to not use the scalar variable, but rather translate it into 3 different steps. This is a bit cumbersome, but works around any timeout issues with the bulk retrieval of snmp tables. This would also verify your thought on the format not being good.

So 3 steps with SNMP walks:

1.3.6.1.2.1.2.2.1.6.1 -> MAC 1

1.3.6.1.2.1.2.2.1.6.2 -> MAC 2

1.3.6.1.2.1.2.2.1.6.3 -> MAC 3

If neither of these work (which would be weird to be honest), i would have to check the pattern manually (so maybe you can give me an XML export of the pattern BUT PLEASE FOR THE LOVE OF GOD MAKE SURE THERE IS NO CUSTOMER DATA IN THERE!). Happy to take a look. I won't fix it for you, but i'd like to guide you. This is a great learning opportunity for me as well as i haven't worked with IoT patterns for a while now (just the "boring" racks and stuff).

Regards

Fabian

[Preston_83]

Hi Fabian,

I wanted to share an update.  I’ve identified the reason my pattern wasn’t being triggered, although I’m still working on a permanent solution.

Using Wireshark packet captures, I found that the production MID Server was only attempting the default publiccommunity string and not the custom community string that has access to the device. After manually configuring an affinity rule for the specific device and MID Server using my custom credential, the pattern triggered and ran as expected.

What’s unclear is why the MID Server continued defaulting to the public string. The public credential was created by the system and has an order of 99,999, whereas my custom credential is set to 50. I also verified that the public credential isn’t listed in any affinity rules.

Next, I need to determine the best way to consistently enforce the use of my custom credential — either by adjusting the selection behavior or by defining affinity rules for 2,000+ IP cameras across multiple VLANs and MID Server clusters.

Let me know if you’ve encountered anything similar or have any recommendations.

Thanks