The CreatorCon Call for Content is officially open! Get started here.

christopher_tho
ServiceNow Employee
ServiceNow Employee

This is in regards to the legacy CI identifiers used prior to the Geneva release and without Service Mapping. For information on how to handle this scenario using the new CI identifiers, please reference the second part of this series: Discover multiple CIs with the same serial number - Part II

A common issue that seems to occur is if you have some virtual devices or cloned devices that have the same Serial Number, Discovery is unable to differentiate these devices and instead will either skip creating some of these CI's or will override currently existing records.

 

For example, let's say I have 2 AIX LPAR's that have these values.

 

1)

Name: Server 1

IP: 10.10.10.1

Serial Number: abc123

 

2)

Name: Server 2

IP: 10.10.10.2

Serial Number: abc123

 

If we scan "Server 1" first, we will create a new AIX server record with its values.

It will look something like below.

Screen Shot 2015-04-28 at 1.12.44 PM.JPG

However, if we then scan "Server 2", what will happen is that Discovery will search the CMDB based on the Serial Number and find that we have "one exact match" already in our database and it will just update the CI record it matched against with the new Name and IP values we just discovered.

Screen Shot 2015-04-28 at 1.11.04 PM.JPG

Thus, we will no longer have any reference to the "Server 1" device information.

 

The reason this occurs is because by default, we assume that the Serial Number of any device that we scan is a unique identifier for these devices.

 

If you look in the "CI Identifiers" table (located under "Discovery Definition > CI Identification > Identifiers) you will see that with the exception of a few special type of Identifiers (such as ESX UUID and Storage Server), the first Identifiers that will be used against most types of devices are " Serial Number Table & Class Name" and "Serial Number & Class Name."

 

Using the Serial Number as the main unique identifier is preferred because it's more likely that other values like Name or IP Address can be changed or are not unique.

 

For example, if a company uses dynamic IP address assignment where a device's IP may change regularly or if a user decides to change the name of their computer, these actions will alter certain values in subsequent Discovery scans of these machines. In these cases, while the IP and/or name may be getting changed, we still want to associate this device when scanned to the same CI record and because the serial number typically is not changed, this is why we use this primarily to identify what CI record to update.

 

However, in this scenario, where we actually have separate devices we are scanning with this same serial number value, this behavior will not work. Overall, the best way to address this issue is to be able to modify the duplicated serial numbers so that you don't have any of these in your environment. However, this may not be a feasible solution is some cases.

 

So, as an alternative to address this type of issue, there are some possible "workarounds" that you can implement. Now, to be clear, none of these suggestions are an "absolute" way to fix this issue, because there are a lot of other varying factors to consider as I will mention below.

 

3 Ways to Discover Multiple CIs with duplicate serial numbers:

 

Create the CI Records Manually

If you have an environment where you know ahead of time what servers may be having these duplicate serial numbers, an easy way to workaround this issue is to create records manually for these CI's that can be referenced when scanning these devices later.

 

Using my initial example for reference, if I were to manually create 2 AIX server records in my CMDB for these 2 servers, what will happen when we scan these devices is that because we won't find "one exact match" with the Serial Number (because we will have 2 records with this same value), the "Serial Number" Identifiers will be skipped, and then we will proceed to try to use other values that may be unique, for example the "Name".

 

So, as long as the name is unique for these devices, then we can make the exact match based on this and update the appropriate CI records accordingly.

 

Now, with this possible solution, there are several factors to consider.

 

1) Along with creating the CI records and putting the Serial Number in the CI records themselves, you will also need to create corresponding records in the "cmdb_serial_number" table as well. This is because for the "Serial Number Table & Class Name" Identifier, we also check for the serial number we find in Discovery with records in this cmdb_serial_number table that link to the corresponding CI records. So, for each CI with this same Serial Number you create, you also need to create a matching "cmdb_serial_number" record as well.

 

2) While this may be effective if you only have a few devices where this issue is occurring, this may be more difficult if you have hundreds of these devices with the same serial number or if the number is unknown.

 

3) As mentioned, with this process we will skip over the "Serial Number" identifiers and use other identifiers like "Name and Class Name."

 

However, if the names of these devices are not unique as well, or if they could be changed, then this presents other issues as well.

 

Deactivate the "Serial Number" Identifiers

Similar to the behavior mentioned from suggestion 1, by deactivating the "Serial Number Table & Class Name" and "Serial Number & Class Name" identifiers, this will cause us to rely on using another Identifier to be able to distinguish CI's, such as "Name & Class Name" or "Network" or even some others in the CI Identifiers table that are inactive by default.

 

This is a little more helpful in the case of the scenario where you have a large number or unknown amount of these devices, however, you still have to be cautious that you need to make sure that ALL your devices (not only just these ones with the same serial number) have a unique name or IP address or whatever you decide to use to try to identify these devices uniquely.

 

Plus, this will affect other device types as well, so even if you have just have this issue with some AIX servers, deactivating these Identifiers will also affect how we are identifying all your Solaris Servers, Windows Servers, Routers, etc. also.

 

Modify the "Serial Number" Identifiers

Of these suggestions, this is probably the most effective one. For this method, you will need to have some understanding of your environment in that you will need to know what type of devices in your environment are having this issue.

 

For example, if this issue is only for some of your AIX servers, then we only need to make a simple modification in the Serial Number Identifiers to essentially skip over using them if scanning AIX servers, but still allow us to use them for other types of devices as normal.

 

To do this, we need to modify the query in the "Serial Number" identifiers that searches for existing records.

 

For example, in the "Serial Number Table & Class Name" Identifier, you would add an additional line to the query as such.

 

CURRENT:

var gr = new GlideAggregate('cmdb_serial_number');

gr.addQuery('valid', 'true');

gr.addQuery('absent', 'false');

gr.query();

Screen Shot 2015-04-28 at 1.22.15 PM.JPG

 

NEW:


var gr = new GlideAggregate('cmdb_serial_number');

gr.addQuery('valid', 'true');

gr.addQuery('absent', 'false');

//Added line below

gr.addQuery('cmdb_ci.sys_class_name', '!=', 'cmdb_ci_aix_server');

gr.query();

Screen Shot 2015-04-28 at 1.23.30 PM.JPG

 

You may also need to add a similar line in the "Serial Number & Class Name" identifier as well, where instead of "cmdb_ci.sys_class_name", you would just need to use "sys_class_name", since we are querying the CI records themselves.

 

**One recommendation I do have is that instead of modifying the out-of-box records, you may want to make a copy of these records, and then make your changes on the copied records and just deactivate the OOB records, this way if we later update these identifiers in a future version, you can still get the updates on the out-of-box records instead of them being skipped over because you made customizations on them.**

 

Again, all we are doing is just skipping to use these identifiers for the potential duplicates that may occur, but with this method, we will at least allow these identifiers to be used if you are scanning the other types of devices like the Windows servers, Routers, etc.

 

Also, another benefit is that you can modify these identifier queries even more specifically if you have more details.

 

For example, if you know the exact serial numbers that can have this issue, you can modify the query to only not check against that specific serial number instead of using the entire class, and in this case you will still be able to use these Identifiers for other devices in the same class as long as they don't have the serial number you have specified in the query.

 

Now, there are still some drawbacks with this method. Let's say, for instance, that later on you do change the name or IP address of one of these records with the duplicate serial number. In the next scan of this device, instead of us being able to identify this as the same CI, we may instead see this as a separate device (since we would have a different name and/or IP address) and therefore may create a new CI record when we shouldn't be.

 

So, while none of these methods are an "absolute" solution for this issue, I hope this can help to at least provide you with some ways to be able to at least assist if you have these duplicate serial number devices on your environment.

 

See also, Discovery and MID Server Resources for additional context and assistance with troubleshooting.

 

If you have some other methods or ways you think may be better to handle this scenario, I would love to hear other suggestions as well.

21 Comments
robpickering
ServiceNow Employee
ServiceNow Employee

Chris, great post!



This recently came up in my Discovery class, specifically around Cisco Nexus 7000 devices.   These are virtual switches that contain multiple virtual routers.



Ideally, we'd like to discover the physical CI (which is the one with the Serial Number) separate from the individual virtual routers contained in the hardware.   This type of discovery would be akin to the ESXi host and the VMs that it hosts.



Your article has provided some great insight into these Identifier and how we may be able to solve this problem going forward...



I'm planning on modifying the Serial Number identifier not for a specific cmdb_ci class (like 'switch'), but actually skip Serial Number identification for specific Models/Names (e.g. N7K-C7004).



Thanks for the pointer!


rtumma
Kilo Contributor

Hi Robert



I am having same issue with Cisco Nexus 7000 devices(Layer 3 Switch). Can you please provide me some information on how to handle the CISCO Nexus 7700 devices? Is your problem resolved?


robpickering
ServiceNow Employee
ServiceNow Employee

Roy, to be honest, I haven't come back to do what was suggested in the article.   Just haven't had the time.   So my issue is not resolved, though I believe the solution presented would do so.



Essentially, you'd have to check the Model number that was identified, and if it was a N7K you'd skip serial number detection.   This would then, most likely, create individual CIs for the Chassis and then for each virtual router, as there would no longer be an identifier that made Discovery think they were the same device (which technically they are).


brendanaye
Giga Contributor

Great post!   I had the same problem with AIX LPARs in our environment, since they were all showing the common serial numbers for the physical frame.   I ended up going with the os_uuid instead of the systemid.   Note it is only present on AIX 6.1 TL6 SP5 and newer, so I fall back to systemid on older systems.



Here is what I used for my AIX identity probe


if lsattr -El sys0 -a os_uuid >/dev/null 2>/dev/null; then lsattr -El sys0 -a os_uuid|awk '{print $2}'; else lsattr -El sys0 -a systemid | awk '{print $2}'; fi


mikefountain
Kilo Contributor

I've been able to get this to work by adding the following to the "Generic Serial Number" and "Serial Number & Class Name" identifiers:



gr.addQuery('model_number', 'DOES NOT CONTAIN', 'n7k');



And the following in the "Serial Number Table & Class Name"



gr.addQuery('cmdb_ci.model_number', 'DOES NOT CONTAIN', 'n7k');




The 7K switches now fall through to the "Name & Class Name" identifier, where it matches on class=switch, and hostname = vdc/partition name


brendanaye
Giga Contributor

If you are upgrading to Geneva there is a new architecture for the identifiers.   You are able to use the Legacy Identifiers unless you are using ServiceWatch, in which case the new identifiers are required.   The new identifiers do not support   "Name & Class Name" so keep this in mind if you upgrade.   It will be a single named-based identifier on the Hardware table unless you create specific identifiers for each class.


sindhuja5
Giga Contributor

Great Post!


I am kind of new to discovery & working on initial discovery setup for my project.


Looking out for some suggestions regarding network device discovery.



while discovering network devices, the IP address of the device(say   Cisco Catalyst 4507rpluse)


& the location is always overwritten by discovery.



For e.g. on Day 1 a CI is created with IP=xx.xxx.xx.1


During the next discovery schedule the ip is updated as xx.xxx.xx.5



Since the scanned IP's are dynamic,wondering if this is the actual behavior/right way to discover network devices.


Kindly advise.




Thanks in Advance!


Sharon Hobart
Mega Guru

I learned recently that Geneva addresses the issue with AIX serial numbers by appending the LPAR ID to the serial number; hence making it a unique serial number.   You can also port this change of the AIX-Serial Number Probe and Sensor from Geneva to Fuji and it works great and solves that issue with AIX.


shyamsundar1
Giga Contributor

Hi All,



I am going through the same issue now, the name is unique though it is not taking Name and Class name identifier.



Kindly help me in understanding this, if the serial number is same it will skip that identifier and will move to 'Name and Class name identifier' or not?



If not do I need to modify the Serial number table and Class identifier?




Regards,


Sundaresan M


Rakesh16i
Kilo Expert

Hi Chris,



We are using Geneva version, and facing the similar problem while discovering Linux LPARS which have same serial number.



I tried your method 1.


      - Discovered server 1 with 10.x.x.1.... it created a CI.


      - I have manually created server2 with IP 10.x.x.2 and same serial number of server1.


      - updated cmdb_serial_number table with server2 details mapping to serial number.



I ran the discovery with 10.x.x.2, still it is changing the server1 values instead fo updating my Manual CI. Not sure what I have missed. Please help.



Regards,


Rakesh