Overview
Recently I've dealt with a few issues, of the customer, which are mostly related to the relationship between the Server CI, VM Instance / VMWare Instance CI, and ESX Server CI (*For VMWare). Most noticed root causes are data issue(s), different customer expectations, etc so this provoked me to come out with this article to provide a clear understanding of how the relationship(s) are created and also how to debug if things go wrong.
This article addresses customer concerns like:
- "Instantiates::Instantiated by" relationship not getting created between the Guest server (Linux, Windows, etc) and the Virtual Machine Instance.
- "Virtualized by::Virtualizes" relationship not getting created between the ESX server and the Guest server
- DiscoveryAWSRelationshipSensor doesn't reconcile old relationships
- DiscoveryAzureRelationshipSensor doesn't reconcile old relationships
- DiscoveryAWSRelationshipSensor script include creates relationships with multiple/duplicated VM Instance
- Discovery not creating "Virtualized by::Virtualizes" relationship
- a few more...
Understanding How the Relationships are created
For Servers deployed in VMWare
During the VMWare Discovery, as a part of the flow, the VMs are discovered and persisted via the system script include with name - "VCenterVMsSensor". In the script include, the "preWriteVm" function triggers the method - "processVmInstanceOrTemplate" of the "VmwareVmCorrelator" system script include.
The "VmwareVmCorrelator" creates the "Instantiates::Instantiated by" relationship between the VMWare Instance record and the Server CI (Linux, Windows, etc). The most important thing is once the relationship is established, the VM Instance's field - "Guest Reconciled" is updated to "True"
In the same VMWare Discovery, as a part of the flow, the ESX Hosts are discovered and persisted via the system script include with name - "VCenterESXHostsSensor". In the script include, the "fixVirtualizes" function creates the "Virtualized by::Virtualizes" relationship between ESX and the Server CI (Linux, Windows, etc) only if the "Instantiates::Instantiated by" relationship is established between the VMWare Instance record and the Server CI.
Similarly when a Server CI (Windows, Linux, etc.) is discovered then the business rule with the name "Virtual Computer Check" is triggered, which would create the"Instantiates::Instantiated by" relationship between the Server CI and VM Instance record. The most important thing is once the relationship is established, the Server CI's field - "Is Virtual" is updated to "True"
For IP Discovery of the Servers deployed in AWS and Azure (Via Probes)
When the IP Discovery is triggered for that specific Server's IP then along with other probes, depending on the classification triggered, a subset of following probes are triggered to confirm if they're deployed in Cloud
- Windows - Amazon EC2
- Windows - Azure
- Linux - Amazon EC2
- Linux - Azure
The above probes end up in triggering the script include - "AWSRelationshipSensor" and "AzureRelationshipSensor" which are responsible for creating the relationship between the VM Instance and the Server CI.
For IP Discovery of the Servers deployed in AWS and Azure (Via Pattern)
The Patterns - "Linux Server", "Windows OS - Servers" are triggered during the Server CI discovery. The Pattern Pre/Post Script - "Create Relation Between Host To VM Instance" is executed, post-execution of the patterns, thus creating the relationship between the Server CI and the VM Instance.
Things to remember Before Debugging
- Deleting the relationships manually isn't recommended and if done then its purely the data customization which will interpret with the expected behavior during the regular discovery.
- Once the fields ("Is Virtual" for Server CI and "Guest Reconciled" for VM Instance) are updated to "True" or checked (if it's a checkbox) then during the normal discovery run, the flow (based on the scripts) assumes the Server CI (Guest) is reconciled to the VM Instance though the relationships are deleted manually.
- At a given point of time, the relationship between the Server CI and VM Instance CI is 1:1 i.e, Server CI cannot have a relationship with multiple VM Instances same time and vice-versa.
Related Information:
KB0832125: Linux servers hosted in AWS are not identified as virtual machines ("Is Virtual" option is not marked in the CI record)
KB0695229: How does Horizontal Discovery of Cloud-based Servers create Virtualized By::Virtualizes relationship between the servers and the virtual instances?)
KB0818921: Azure Discovery not creating "Virtualized by:: Virtualizes" relationship
Known Issues
ISSUE-1: Multiple virtualized relationships exists between the VMs and one single Server CI
Root Cause: When Server CI deployed in Cloud (AWS, Azure, VMWare) if provided the same name then due the CI Identifier for cmdb_ci_computer is "Name" so the same Server CI is updated all the time and adds the virtualized relationship with various VM Instance records.
Solution: Any below should resolve the issue
- Provide a different name for different Server CIs but not the same name for all Server CIs deployed in the Cloud
- Update the CI Identifier to use another or additional field(s) instead of "Name" alone. Please note this approach is least recommended because it affects the other CIs' discovery as well.
ISSUE-2: I want to delete the multiple invalid virtualized relationships associated with a Server CI apart from the one valid virtualized relation with the VM Instance CI as per the current discovery run.
Root Cause: This issue occurs when ISSUE-1 is encountered.
Solution: Go to the system script include - "DiscoveryAWSRelationshipSensor" (If the Server CI is in AWS) or "DiscoveryAzureRelationshipSensor" (If the Server CI is in Azure) and the method is - "findAndCreateRelationToVM()".
For now, let's go with "DiscoveryAWSRelationshipSensor" and add the below lines before the existing code lines.
// Additinal script for deleting existing "Virtualized by::Virtualizes" relationships
var relGR = new GlideRecord("cmdb_rel_ci");
relGR.addQuery('parent', ciSysId);
relGR.addQuery("type", g_disco_functions.findCIRelationshipType("cmdb_rel_type", "Virtualized by::Virtualizes"));
relGR.addQuery("child", '!=',vmInstanceSysId);
relGR.query();
if (relGR.getRowCount())
relGR.deleteMultiple();
//
// Existing Code line
if (vmGlideRecord.next())
g_disco_functions.createRelationshipIfNotExists(ciSysId, vmGlideRecord, "Virtualized by::Virtualizes");
ISSUE-3: AWS VM is duplicated in the cmdb_ci_vm_instance table and the Server CI isn't relating to the correct VM Instance CI.
Root Cause: First of all same VM instance records existing in the cmdb_ci_vm_instance table, is expected because the IRE (Identification and Reconciliation Engine) itself create the duplicates but updating the "Duplicate Of" field of each CI so the main CI will have the field referenced to nothing and whereas other duplicated CIs have the field referencing to the main CI.
Solution: PRB1406405 [Fixed in Paris]. PRB includes the modification made for the "DiscoveryAWSRelationshipSensor" script include to pick the right VM.
Changed line 63 FROM:
if(relationsGlideRecord.parent.object_id.indexOf(valueToSearch) != -1){
TO:
if(relationsGlideRecord.parent.object_id.indexOf(valueToSearch) != -1 && JSUtil.nil(relationsGlideRecord.parent.duplicate_of)){
ISSUE-4: In Azure, the "Linux - Azure" probe doesn't work properly if the Linux VM is on Static IP Address thus not creating "Virtualized by::Virtualizes" relationship.
Root Cause: In Azure, Linux server is using Static IP address and the Server has no such file as /var/lib/dhcp/dhclient.eth0.leases or /var/lib/dhclient/dhclient-eth0.leases.
Solution: PRB1327557 [Fixed in New York Pacth-8]. Please make sure to add the command - "dmidecode", "grep UUID" SSH commands in the respective Server CI's sudoers configuration file.
ISSUE-5: Relationship between Server CI and VM instance isn't created if the relationship is deleted manually.
Root Cause: Deleting the relationships manually isn't recommended and if done then its purely the data customization which will interpret with the expected behavior during the regular discovery.
Solution: Reset the "Is Virtual" flag of Server CI to "False" and "Guest Reconciled" flag of VM Instance CI to "False" => Rerun the discovery (Either Cloud (IP incase of VMWare or Server's IP Discovery)
Please note if the above-mentioned flags are read-only the please go to the respective dictionary entries and set the "Read Only" to "False" and continue as suggested above.
ISSUE-6: "Virtualized by::Virtualizes" relationship isn't created between the ESX Server and Server CI.
Root Cause: If the "Instantiates::Instantiated by" relationship isn't added relationship between the Server CI and the VM Instance CU then the "Virtualized by::Virtualizes" relationship isn't created.
Solution: Issue-5 is the root cause for this issue so reset the "Is Virtual" flag of Server CI to "False" and "Guest Reconciled" flag of VM Instance CI to "False" => Rerun the VMWare discovery so that the relationships are properly created.
ISSUE-7: Discovery of Solaris Server is not creating relationships with VM Zones and Solaris Virtual Machine Instance CIs
Root Cause: Pattern Pre/Post Script - "Solaris OS - Post Sensor" is responsible for creating the "Instantiates::Instantiated by" relationship (between the Server CI and the Solaris Virtual Machine Instance) and "Virtualized by::Virtualizes" relationship (between the Server CI and the VM Zones). If the "correlation_id" of the Server CI is empty then based on the below check the flow will continue to trigger the ADM probe rather than building the above relationships.
if(!hasZones() && StringUtil.nil(computerGlideRecord.correlation_id)){
// trigger ADM probe
}
else{
// handleTheRelationships;
}
Solution: Please check the "Solaris Server" pattern response via the ECC Queue (Input) of the Discovery Status and confirm if the correlation_id, for the Server CI, is being retrieved successfully or not.
Relationships between VM, Guest and the ESX (as per VMWare): Relationships as per VMWare.png
