How do I create a custom PowerShell Probe to retrieve additional data about ESX Servers?

Michael Zglesze · ‎05-29-2019

Hello Community!

As the Discovery Administrator, I've been tasked with gathering additional attributes about ESX Servers. After weeks of searching the Docs and the Community, and poking around our development instance of ServiceNow, I cannot figure out for the life of me how to do this.

My thoughts were simply to:

Create a new Probe.
Add a Probe Parameter for the PowerShell script.
Create a new Sensor to parse and process the data, then write it to the CMDB.

My brain is fried at this point, so if someone can point me to a resource or give me some step-by-step instructions, that would be fantastic! 🙂

Thanks in advance!

Michael Zgleszewski

Michael Zglesze · ‎08-06-2019

I FIGURED IT OUT!! 😄

I'll share with the community, but the solution is not for the faint of heart!

I've attached a 20-page Word Document with step-by-step instructions, screenshots, links, and code explanations, but below I've supplied the short, short version.

All in all, the solution is to comment out the out-of-the-box functionality where ServiceNow is writing the Serial Number to the ESX Server and Serial Number tables. Then:

Create your own custom Probe and Sensor
Trigger the Probe from the VMWare - vCenter Datacenters Sensor
Create the Sensor/Probe Conditional record/script that actually triggers the custom Probe
Create a MID Server Script Include (which is just a placeholder)
Finally, create a Script Include that launches a PowerShell Probe against the vCenter and processes the results into the CMDB.

The only concern I have about this solution is the passing of the Credentials to the PowerShell script. If someone can suggest a more secure way to do this, it would complete the masterpiece.

Have fun with this! 😄

1. Install the VMware.PowerCLI (and associated) PowerShell Modules on all applicable MID Servers.
2. Delete all entries in the Serial Number [cmdb_serial_number] table for ESX Servers.
3. Create the Probe: VMware - vCenter ESX Serial Numbers

Probe Type: Probe
Cache Results: false
ECC Queue Topic: VMWareProbe
ECC Queue Name: VMWareVCenterESXHostsSerialNumberProbe
Used by Discovery: true
Used by Orchestration: false

4. Create the Sensor: VMware - vCenter ESX Serial Numbers

Reacts to Probe: VMware - vCenter ESX Serial Numbers
Sensor Type: Sensor
Sensor Type: Javascript
Script:

//	Set the vCenter parameter to the IP address of the vCenter being discovered
var vCenter = g_probe.getParameter('source');

//	Set the midServer to be used in the custom PowerShell Probe to be the same as the MID Server processing this Sensor
var midServer = g_probe.getParameter('agent');

//	Set the host that will be executing the custom PowerShell Probe; this will be the local host
var psServer = '127.0.0.1';

//	Call the Script Include that executes the custom PowerShell Probe
//	This Script Include also processes the returned data as if it was the Sensor
var getESX_SN = new VMWareVCenterESXHostsSerialNumberProbe(vCenter, midServer, psServer);

5. Modify the Sensor, VMWare - vCenter Datacenters, to Trigger the new ESX Serial Number Probe.
6. Create a record in the "Conditional probes triggered by sensor" [discovery_sensor_probe_conditional] table to allow the new ESX Serial Number Probe to be Triggered.

Probe: VMware - vCenter ESX Serial Numbers
Sensor: VMWare - vCenter Datacenters
Condition Script:

// This script gets run to prepare parameters for triggered probes
/*jshint -W025 */	//	Remove JSHint warning:  "Missing name in function declaration"

function (parms, data) {
	parms.agent = g_probe.getParameter('agent');
	parms.source = g_probe.getParameter('source');
	parms.priority = 1;
	parms.topic = 'VMWareProbe';
	parms.queue = 'output';
	
	var rightNow = new GlideTime();
	var tenSecondsAgo = new GlideTime();
	tenSecondsAgo.subtract(10000);

	var grECC = new GlideRecord('ecc_queue');
	grECC.addQuery('name', 'VMWareVCenterESXHostsSerialNumberProbe');
	grECC.addQuery('sys_created_on', '>=', tenSecondsAgo).addOrCondition('sys_created_on', rightNow);
	grECC.query();

	while(grECC.next()) {
		if(grECC.source == g_probe.getParameter('source')) {
			return false;
		}
	}
	
	if(grECC.hasNext()) {
		gs.sleep(60000);
		return true;
	} else {
		return true;
	}
}

7. In the MID Server Script Include, VMWarevCenterESXHostsProbe, comment out the portion of the code that writes the Serial Number to the ESX Server table [cmdb_ci_esx_server]:

if (oi) {
	oi.forEach(function(id) {
		var fieldname,
		val = id.value,
		key = id.IdentifierType;

		if (val && (key.key == 'ServiceTag'))
			host.serial_number = val;
	});
}

8. In the Script Include, VCenterESXHostsSensor, comment out the portion of the code that writes the Serial Number to the Serial Number table [cmdb_serial_number]:

var serialNumber = {
	cmdb_ci: esx,
	serial_number: esx.serial_number,
	serial_number_type: 'chassis',
	valid: SncSerialNumber().isValid(esx.serial_number)
};

serialNumbers.push(serialNumber);

9. Create the MID Server Script Include that is called by the custom ESX Serial Number Probe: VMWareVCenterESXHostsSerialNumberProbe

Name: VMWareVCenterESXHostsSerialNumberProbe
Script:

var VMWareVCenterESXHostsSerialNumberProbe = Class.create();

(function() {
	VMWareVCenterESXHostsSerialNumberProbe.prototype = Object.extendsObject(AVMWareProbe, {

		process : function() {
			//	This Probe is really just a placeholder.
			//	The only thing it does is complete successfully and get responded to by the Sensor.
		},

		type : 'VMWareVCenterESXHostsSerialNumberProbe'
	});
})();

10. Create the Script Include that the custom ESX Serial Number Sensor calls: VMWareVCenterESXHostsSerialNumberProbe

Name: VMWareVCenterESXHostsSerialNumberProbe
Client Callable: true
Script:

/*jshint multistr: true */  //  Prevents the multi-line string warning
/*jshint -W083 */ //  Prevents the "Don't create a function within a loop" warning

var VMWareVCenterESXHostsSerialNumberProbe = Class.create();

VMWareVCenterESXHostsSerialNumberProbe.prototype = {
  initialize: function(vCenter, midServer, psServer) {
    //  Create a new instance of DiscoveryFunctions to be used to initialize the DiscoveryLogger
    var discoFunctions = new DiscoveryFunctions();

    //  Create a new instance of the DiscoveryLogger so that Error, Warnings, and Information messages can be written to the Discovery Log
    var discoLog = new DiscoveryLogger(discoFunctions.getStatusRecord().sys_id);

    //  Set the variables to be used in the Logging functions
    var loggerSource = 'VMware - vCenter ESX Serial Number';
    var loggerSensor = null;
    var loggerCI = this.getCmdbCi();
    
    //  Get the stored vCenter Credentials
    var credProvider = new sn_cc.StandardCredentialsProvider();
    var credentials = credProvider.getCredentials(['VMware']);

    var objCred = {};
    var domain = '';

    for(var h = 0; h < credentials.length; h++) {
      domain = ((credentials[h].getAttribute('user_name').indexOf('DEV_DOMAIN\\') >= 0) ? ('DEV_DOMAIN') : ((credentials[h].getAttribute('user_name').indexOf('PROD_DOMAIN\\') >= 0) ? ('PROD_DOMAIN') : ('')));
      
      objCred[domain] = {
        u : credentials[h].getAttribute('user_name'),
        p : credentials[h].getAttribute('password')
      };
    }

    //  Store the Powershell Script that is to be executed in a variable
    var thePS_Script = "# Import the VMware Module \
    $theImport = Import-Module -Name VMware.VimAutomation.Core \
    \
    # Define the vCenter Host being discovered and it's domain \
    $theHost = '" + vCenter + "' \
    $theDomain = [System.Net.Dns]::GetHostByAddress($theHost).Hostname.split('.') \
    \
    # Define Connection Credentials \
    if($theDomain[1] -eq 'PROD_DOMAIN' -or $theDomain[1] + '.' + $theDomain[2] -eq 'dns.domain') { \
      $theCredUsername = '" + objCred.PROD_DOMAIN.u + "' \
      $theCredPassword = '" + objCred.PROD_DOMAIN.p + "' \
    } elseif($theDomain[1] -eq 'DEV_DOMAIN' -or $theDomain[1] -eq 'LAB') { \
      $theCredUsername = '" + objCred.DEV_DOMAIN.u + "' \
      $theCredPassword = '" + objCred.DEV_DOMAIN.p + "' \
    } \
    \
    # Connect to vCenter being discovered \
    $theConnection = Connect-VIServer -Force -Server $theHost -User $theCredUsername -Password $theCredPassword \
    \
    # Print all ESX Server Serial Numbers \
    $theCustomProperty = New-VIProperty -ObjectType VMHost -Name SerialNumber -Value { (Get-EsxCli -VMHost $Args[0] -V2).hardware.platform.get.Invoke().SerialNumber } \
    Get-VMHost | Where-Object  {($_.ConnectionState -eq \"Connected\" -or $_.ConnectionState -eq \"Maintenance\")} | Select Name,@{n='HostUUID';e={$_.ExtensionData.hardware.systeminfo.uuid}},SerialNumber | Format-Table -AutoSize -HideTableHeaders \
    \
    # Clear Authentication Variables \
    $theCred = $null \
    \
    # Disconnect from vCenter \
    Disconnect-VIServer * -Confirm:$false";
    
    var xmlDocPayloadOutput = new XMLDocument("<parameters/>");

    var theElement = xmlDocPayloadOutput.createElement("parameter");
    xmlDocPayloadOutput.setCurrent(theElement);
    xmlDocPayloadOutput.setAttribute("name", "skip_sensor");
    xmlDocPayloadOutput.setAttribute("value", "true");

    xmlDocPayloadOutput.setCurrent(xmlDocPayloadOutput.getDocumentElement());
    theElement = xmlDocPayloadOutput.createElement("parameter");
    xmlDocPayloadOutput.setCurrent(theElement);
    xmlDocPayloadOutput.setAttribute("name", "probe_name");
    xmlDocPayloadOutput.setAttribute("value", "VMware - vCenter ESX Serial Number");

    xmlDocPayloadOutput.setCurrent(xmlDocPayloadOutput.getDocumentElement());
    theElement = xmlDocPayloadOutput.createElement("parameter");
    xmlDocPayloadOutput.setCurrent(theElement);
    xmlDocPayloadOutput.setAttribute("name", "script.ps1");
    xmlDocPayloadOutput.setAttribute("value", thePS_Script);
    
    xmlDocPayloadOutput.setCurrent(xmlDocPayloadOutput.getDocumentElement());
    theElement = xmlDocPayloadOutput.createElement("parameter");
    xmlDocPayloadOutput.setCurrent(theElement);
    xmlDocPayloadOutput.setAttribute("name", "vCenter");
    xmlDocPayloadOutput.setAttribute("value", vCenter);
    
    //  Launch JavascriptProbe to execute the Powershell script
    var psProbe = new GlideRecord('ecc_queue');
    psProbe.initialize();
    psProbe.agent = midServer;
    psProbe.topic = 'Powershell';
    psProbe.name = 'VMware - vCenter ESX Serial Number';
    psProbe.source = psServer;
    psProbe.queue = 'output';
    psProbe.state = 'ready';
    psProbe.payload = xmlDocPayloadOutput.toString();
    var eccOutputSysID = psProbe.insert();
    
    //  Query for the Input ECC Queue record that corresponds to the Output ECC Queue record launched above
    var eccRecordFound = false;
    
    var grECC_Response = new GlideRecord('ecc_queue');
    grECC_Response.addQuery('queue', 'input');
    grECC_Response.addQuery('response_to', eccOutputSysID);
    
    //  We've got to wait until the ECC Queue Output record in done processing, so check for the existance of the Input record every one second.
    for(var i = 0; (i < gs.getProperty('com.snc.integration.powershellprobe.maxWaitTimeForEccQueue') && (eccRecordFound == false)); i++) {
      gs.sleep(1000);
      grECC_Response.query();
      eccRecordFound = ((grECC_Response.next()) ? (true) : (false));
    }
    
    //  When the ECC Queue Input record is available, grab the XML Payload
    var theGlideSysAttachment = new GlideSysAttachment();
    var grECC_Response_Payload = ((grECC_Response.payload != '<see_attachment/>') ? (grECC_Response.payload) : (theGlideSysAttachment.get(grECC_Response, 'payload')));
    
    var xmlDocPayloadInput = new XMLDocument(grECC_Response_Payload);
    var xmlDocPayloadInputString = xmlDocPayloadInput.toString();
    var xmlDoc2PayloadInput = new XMLDocument2();
    xmlDoc2PayloadInput.parseXML(xmlDocPayloadInputString);
    
    var objXML_Payload = {
      output  : '',
      error : ''
    };
    
    if(eccRecordFound) {
      grECC_Response.setWorkflow(false);
      grECC_Response.state = 'processed';
      grECC_Response.update();
      
      if(JSUtil.notNil(xmlDoc2PayloadInput.getNodeText('//results/result/output'))) {
        objXML_Payload.output = xmlDoc2PayloadInput.getNodeText('//results/result/output').toString();
      } else {
        objXML_Payload.output = null;
      }
      
      if((JSUtil.notNil(xmlDoc2PayloadInput.getNodeText('//results/result'))) || (JSUtil.notNil(xmlDoc2PayloadInput.getNodeText('//results/result/error')))) {
        objXML_Payload.error = xmlDoc2PayloadInput.getNodeText('//results/result').toString() + '\n' + xmlDoc2PayloadInput.getNodeText('//results/result/error').toString();
      } else {
        objXML_Payload.error = null;
      }
    }
    
    //  Only continue processing the output if there are no errors
    if(objXML_Payload.error) {
      discoLog.warn(objXML_Payload.error, loggerSource, loggerSensor, loggerCI);
      return;
    } else if(!objXML_Payload.output) {
      discoLog.warn('ERROR:  No data was returned from the ESX Serial Number probe.', loggerSource, loggerSensor, loggerCI);
      return;
    } else {
      var arrayData = [];
      var arrayParts = [];
      var arrayESX = [];
      
      arrayData = objXML_Payload.output.split('\n');
      
      //  Remove all of the extra, empty lines in arrayData
      arrayData = arrayData.filter(function(theElement) { if(JSUtil.notNil(theElement)) {return true;} });
      
      //  Remove all whitespace with one comma so that it can be parsed into three tokens
      arrayData = arrayData.map(function(theElement) { return theElement.replace(/\s+/g, ','); });
      
      for(var j = 0; j < arrayData.length; j++) {
        arrayParts = arrayData[j].split(',');
        
        var objESX = {
          server    : ((JSUtil.notNil(arrayParts[0]).toString()) ? (arrayParts[0].toString().toUpperCase().match(/^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|^(\w+)/)[0]) : ('')),
          uuid    : ((JSUtil.notNil(arrayParts[1]).toString()) ? (arrayParts[1].toString().toUpperCase()) : ('')),
          sn      : ((JSUtil.notNil(arrayParts[2]).toString()) ? (arrayParts[2].toString().toUpperCase()) : ('')),
          printInfo : function() { return 'Server:\t\t' + this.server + '\nSerial Number:\t' + this.sn + '\nUUID:\t\t' + this.uuid; }
        };
        
        if((JSUtil.notNil(objESX.server)) && (JSUtil.notNil(objESX.sn))) {
          arrayESX.push(objESX);
        }
      }
      
      //  Sort the array by server name so that the output is friendlier, and troubleshooting is easier.
      arrayESX.sort(function (a, b) {var strA = a.server.toString().toUpperCase(); var strB = b.server.toString().toUpperCase(); return ((strA > strB) ? (1) : ((strA < strB) ? (-1) : (0))); });
      
      //  Search for duplicate UUID an SN values and push them onto the appropriate array
      for(var y = 0; y < arrayESX.length; y++) {
        for(var z = 0; z < arrayESX.length; z++) {
          if(y == z) {
            //  If the same index is being compared, then the values are the same, but it is not a duplicate; it is itself.  :-D
            continue;
          } else {
            //  If the UUID values match, this is a duplicate; create the deleteMe key and set the value to true
            if(arrayESX[y].uuid == arrayESX[z].uuid) {
              if(!arrayESX[y].hasOwnProperty('deleteMe')) {
                arrayESX[y].deleteMe = true;
              }
              
              if(!arrayESX[z].hasOwnProperty('deleteMe')) {
                arrayESX[z].deleteMe = true;
                discoLog.warn('\n\nThese Configuration Items cannot be updated because VMware vCenter states the following ESX Servers have the same UUID:\n\n' + arrayESX[y].printInfo() + '\n\n' + arrayESX[z].printInfo(), loggerSource, loggerSensor, loggerCI);
              }
            }
            
            //  If the Serial Number values match, this is a duplicate; create the deleteMe key and set the value to true
            if(arrayESX[y].sn == arrayESX[z].sn) {
              if(!arrayESX[y].hasOwnProperty('deleteMe')) {
                arrayESX[y].deleteMe = true;
              }
              
              if(!arrayESX[z].hasOwnProperty('deleteMe')) {
                arrayESX[z].deleteMe = true;
                discoLog.warn('\n\nThese Configuration Items cannot be updated because VMware vCenter states the following ESX Servers have the same Serial Number:\n\n' + arrayESX[y].printInfo() + '\n\n' + arrayESX[z].printInfo() + '\n\n', loggerSource, loggerSensor, loggerCI);
              }
            }
          }
        }
      }
      
      //  Remove the duplicates from arrayESX so that we can process the non-duplicates
      arrayESX = arrayESX.filter(function(theElement) { return !(theElement.hasOwnProperty('deleteMe')); } );
      
      //  Loop through the ESX Server array and update the ESX Server table [cmdb_ci_esx_server] and the 
      //  Serial Number table [cmdb_serial_number] with the correct information, if necessary
      for(var x = 0; x < arrayESX.length; x++) {
        var grCorrectESX_Asset = new GlideRecord('alm_hardware');
        grCorrectESX_Asset.addQuery('serial_number', arrayESX[x].sn);
        grCorrectESX_Asset.query();
        
        grCorrectESX_Asset.setWorkflow(false);
        
        var grESX_Server = new GlideRecord('cmdb_ci_esx_server');
        
        grESX_Server.addQuery('u_active', true);
        grESX_Server.addQuery('correlation_id', arrayESX[x].uuid);
        grESX_Server.query();
        
        grESX_Server.setWorkflow(false);
        
        if(grESX_Server.next()) {
          if(grESX_Server.serial_number != arrayESX[x].sn) {
            //  Remove the Asset from the CI so that the Asset's serial number does not get updated improperly
            grESX_Server.asset = '';
            grESX_Server.update();
            
            //  Remove the CI from the Asset so that the Asset's serial number does not get updated improperly
            if(grCorrectESX_Asset.next()) {
              grCorrectESX_Asset.cmdb_ci = '';
              grCorrectESX_Asset.update();
            }
            
            //  Update the CI's Serial Number and assign the correct Asset
            grESX_Server.serial_number = arrayESX[x].sn;
            grESX_Server.asset = grCorrectESX_Asset.sys_id.toString();
            grESX_Server.update();
            
            //  Assign the CI to the correct Asset
            if(grCorrectESX_Asset.next()) {
              grCorrectESX_Asset.cmdb_ci = grESX_Server.sys_id.toString();
              grCorrectESX_Asset.update();
            }
          } else {
            //  Check to see if an asset is attached.  If so, is it the correct Asset?
            if(grCorrectESX_Asset.next()) {
              if(JSUtil.notNil(grESX_Server.asset)) {
                if(grESX_Server.asset != grCorrectESX_Asset.sys_id.toString()) {
                  grESX_Server.asset = grCorrectESX_Asset.sys_id.toString();
                }
                
                if(grCorrectESX_Asset.cmdb_ci != grESX_Server.sys_id.toString()) {
                  grCorrectESX_Asset.cmdb_ci = grESX_Server.sys_id.toString();
                }               
              } else {
                grESX_Server.asset = grCorrectESX_Asset.sys_id.toString();
                
                if(grCorrectESX_Asset.cmdb_ci != grESX_Server.sys_id.toString()) {
                  grCorrectESX_Asset.cmdb_ci = grESX_Server.sys_id.toString();
                }
              }
            }
          }
          
          //  Add the Serial Number and the UUID to the Serial Number [cmdb_serial_number] table, if not already there.
          var grSN = new GlideRecord('cmdb_serial_number');
          
          grSN.addQuery('cmdb_ci', grESX_Server.sys_id);
          grSN.addQuery('serial_number_type', 'chassis');
          grSN.addQuery('serial_number', arrayESX[x].sn);
          grSN.query();
          
          //  If the Serial Number does not already exist in the Serial Number [cmdb_serial_number] table, then create an entry for it.
          //  If the Serial Number already exists, then ensure that the Absent value is set to false.
          if(grSN.next()) {
            //  Prevent any Business Rules from running before updating the Serial Number record
            grSN.setWorkflow(false);
            
            if(grSN.absent != false) {
              grSN.absent = false;
              grSN.update();
            }
          } else {
            var grNewSN = new GlideRecord('cmdb_serial_number');
            
            grNewSN.initialize();
            
            //  Set the Configuration Item
            grNewSN.cmdb_ci = grESX_Server.sys_id;
            
            //  Set the Serial Number
            grNewSN.serial_number = arrayESX[x].sn;
            
            //  Set the Serial Number Type
            grNewSN.serial_number_type = 'chassis';
            
            //  Set the Absent value to false
            grNewSN.absent = false;
            
            //  Set the Valid value to true
            grNewSN.valid = true;
            
            //  Prevent any Business Rules from running before inserting the new Serial Number record
            grNewSN.setWorkflow(false);
            
            //  Insert the new Serial Number record
            grNewSN.insert();
          }
          
          var grUUID = new GlideRecord('cmdb_serial_number');
          
          grUUID.addQuery('cmdb_ci', grESX_Server.sys_id);
          grUUID.addQuery('serial_number_type', 'UUID');
          grUUID.addQuery('serial_number', arrayESX[x].uuid);
          grUUID.query();
          
          grUUID.setWorkflow(false);
          
          //  If the UUID does not already exist in the Serial Number [cmdb_serial_number] table, then create an entry for it.
          //  If the UUID already exists, then ensure that the Absent value is set to false.
          if(grUUID.next()) {
            //  Prevent any Business Rules from running before updating the Serial Number record
            grUUID.setWorkflow(false);
            
            if(grUUID.absent != false) {
              grUUID.absent = false;
              grUUID.update();
            }
          } else {
            var grNewUUID = new GlideRecord('cmdb_serial_number');
            
            grNewUUID.initialize();
            
            //  Set the Configuration Item
            grNewUUID.cmdb_ci = grESX_Server.sys_id;
            
            //  Set the Serial Number
            grNewUUID.serial_number = arrayESX[x].uuid;
            
            //  Set the Serial Number Type
            grNewUUID.serial_number_type = 'UUID';
            
            //  Set the Absent value to false
            grNewUUID.absent = false;
            
            //  Set the Valid value to true
            grNewUUID.valid = true;
            
            //  Prevent any Business Rules from running before inserting the new Serial Number record
            grNewUUID.setWorkflow(false);
            
            //  Insert the new Serial Number record
            grNewUUID.insert();
          }
          
          grESX_Server.update();
          grCorrectESX_Asset.update();
        }
      }
    }
  },

  type: 'VMWareVCenterESXHostsSerialNumberProbe'
};

View solution in original post

VaranAwesomenow · ‎08-06-2019

Nice solution, may be you can post it on share !

Checkout Varan awesomenow youtube channel for more learning videos!

Michael Zglesze · ‎09-11-2019

Hello again, All!

Quick note about my solution...

I found after I implemented this that for some reason, the code I used to match up the Asset to the CI created duplicate Assets and put those on the CIs instead of just putting the matching Asset on the CI.

I would recommend removing the entire section of code that deals with the Assets so that this doesn't happen in your instance.

My appologies. In general, I believe that Discovery shouldn't really be touching Assets, anyway, but at my company, there were some things to be fixed with regards to ESX Servers, so I put it in there.

Again, remove the code dealing with Assets, as it created a boat-load of duplicate Assets in my instance.

Claire Ashdown · ‎06-14-2021

Hello Michael,

Thank you for this wonderfully documented solution for running PowerShell via connect-viserver. Have you found a more secure way of handling the credential situation? I have a business requirement to run PowerShell to retrieve custom attributes during vCenter discovery, but we use an external credential manager, so I would not be able to hard-code those values into $theCredUsername and $theCredPassword. Is this something you have run into as well over the last two years?

Many thanks,

Claire