Integrate with Microsoft Intune - Mapping data with RTE/IRE

Kieran Anson · ‎12-13-2021

< Previous Article		Next Article >
IntegrationHub Data Retrieval		Scheduled Data Load

Introduction

Rather than transforming data using the somewhat legacy transform maps, this article will talk through using RTE & the IntegrationHub ETL that walks us through the process of robustly transforming data destined for the CMDB. This allows for re-assurances that the inbound data is clean and reconciles correctly when performing upsert operations.

New Discovery Source

Creating a new discovery source is optional but will aid in knowing where a record originated from and also allow for IRE rules to be setup to restrict what data sources can perform what actions. To create a new discovery source, use the script below in scripts background.

new CMDBDataSourceUtil().addDataSource("Microsoft Intune");

Basic Details

From the application navigator search for IntegrationHub ETL and create a new record
Set the fields as below, changing the data source to the one you created earlier

Select Save followed by Mark Complete

Preview & Prepare

There are a number of steps to complete here, ensure you follow each step carefully to achieve the expected result.

Lookup Assigned To

Select New Transform and Use Source Column
Set the transform type to User Lookup
Set the output column name to assigned_to
Set the user name to emailAddress
Use the + icon to add another row and set the email to emailAddress
Select Apply

Cleanse MAC Address

Select New Transform and Use Source Column
Set the transform type to Cleanse MAC Address
Set the output column name to cleansed_wifi_mac
Set the mac address as wiFiMacAddress
Select Apply

Cleanse Operating System

Select New Transform and Use Source Column
Set the transform type to Cleanse Operating System
Set the output column name to cleansed_os
Set the mac address as operatingSystem
Select Apply

Cleanse Serial Number

Select New Transform and Use Source Column
Set the transform type to Cleanse Serial Number
Set the output column name to cleansed_systemserialnumber
Set the serial number to serialNumber
Select Apply

Cleanse Hardware Model

Select New Transform and Use Source Column
Set the transform type to Cleanse Hardware Model
Set the output column name to cleansed_hardware_model_concat
Set the company name as manufacturer
Set the model to model
Select Apply
Select New Transform and Use Source Column
Set the transform type to Split
Set the input column to the one we just created
Set the seperator to 3 pipe characters |||
Select Generate Output Columns
Set the field values to cleansed_manufacturer_sys_id, cleansed_manufacturer_name, cleansed_model_sys_id, cleansed_model_name
Select Apply

Disk Space Constants

Select New Transform and Set Fixed Value Column
Set the output column name to storage_units_in
Set the description to Disk Space Units In
Set the value to B
Select Apply
Repeat the process with a name of storage_units_out, value of GB and description of Disk Space Units

Disk Storage

Select New Transform and Use Source Column
Set the transform type to Extract & Scale By Units
Set the output column name to disk_space_gb
Set the input values to totalStorageSpaceInBytes, storage_units_in, storage_units_out
Select Apply

Serial Number Type

Select New Transform and Use Source Column
Set the transform type to Script Operation
Set the output column name to systemserialnumber_type
Set the input column to cleansed_systemserialnumber
Use the script noted below
Select Apply

(function(batch, output) {
   for (var i = 0; i < batch.length; i++) {
      var input = batch[i].input;
      output[i] = input ? "system" : "";
   }
})(batch, output);

Serial Number Validity

Select New Transform and Use Source Column
Set the transform type to Script Operation
Set the output column name to systemserialnumber_valid
Set the input column to cleansed_systemserialnumber
Use the script noted below
Select Apply

(function(batch, output) {
 for (var i = 0; i < batch.length; i++) {
  var input = batch[i].input; 
    output[i] = input ? "true" : "";
 }
})(batch, output);

Cleansed Device Name

Select New Transform and Use Source Column
Set the transform type to Multiple Input Script
Set the output column name to cleansed_device_name
Set the input columns to cleansed_systemserialnumber, deviceName, cleansed_wifi_mac, cleansed_model_name
Use the script noted below
Select Apply

(function(batch, output) {
   for (var i = 0; i < batch.length; i++) {
      var mac = batch[i].cleansed_wifi_mac;
      var cmodel = batch[i].cleansed_model_name;
      var serial = batch[i].cleansed_systemserialnumber;
      var device_name = batch[i].deviceName;
      var cleansed_device_name = "";
      
      if(serial & cmodel)
      cleansed_device_name = serial + "-" + cmodel;
      else if (serial)
      cleansed_device_name = serial;
      else if (mac)
      cleansed_device_name = mac;
      else cleansed_device_name = device_name;
      
      output[i] = cleansed_device_name;
   }
})(batch, output);

Network Source Native Key

This value is used by IRE to understand the generated data and create a relationship between the computer & network adapter

Select New Transform and Use Source Column
Set the transform type to Concatentation
Set the output column name to network_snk
Set the input columns to id and cleansed_wifi_mac
Set the concat value as a single pipe character |
Select Apply

Create Software Instance

Select New Transform and Use Source Column
Set the transform type to Create Software Instance Name
Set the output column name to software_instance_name
Set the hardware name to cleansed_device_name
Set the software name to cleansed_os
Set the software version to osVersion
Select Apply

OS Native Key

Select New Transform and Use Source Column
Set the transform type to Multiple Input Script
Set the output column name to os_native_key
Set the input columns to cleansed_os and osVersion
Use the script noted below
Select Apply

(function(batch, output) {
   for (var i = 0; i < batch.length; i++){
      var os_name = batch[i].cleansed_os;
      var os_version = batch[i].osVersion;
      var os_native_key = "";
      
      if (os_version)
      os_native_key = os_name + "_:::_" + os_version;
      else
      os_native_key = os_name + "_:::_NULL";
      
      output[i] = os_native_key;
   }
})(batch, output);

OS Instance Native Key

Select New Transform and Use Source Column
Set the transform type to Concatentation
Set the output column name to os_instance_native_key
Set the input columns to os_native_key and id
Set the concat value as a single pipe character |
Select Apply

Select the blue Mark as Complete button

Map Data to CMDB

Now time to map the data to where we want it in the CMDB. To get started open Select CMDB Classes to Map Source Data

Create Computer CI

Select Add Class
Select Computer and save
Select Set Up Mapping
Under Source Native Key, drag the id field to the Source Column. You can also use the pill picker to map the value
Set the name source column to cleansed_device_name
Set the serial number source column to cleansed_systemserialnumber
For Serial Number Lookup 1 set the serial number to cleansed_systemserialnumber & type to systemserialnumber_type
Use the Add Serial Number Attribute to add the column valid and map systemserialnumber_valid
For Network Adapter Lookup 1 set the mac address and name to cleansed_wifi_mac
Use the add network adatper attribute button to add Source Native Key and map network_snk
At the top of the form, select add attribute and add the following (note you can select multiple fields before pressing save): disk space, manufacturer, operating system, OS version, model ID, Assigned to, Chassis type
Map disk_space_gb to disk space
Map cleansed_manufacturer_sys_id to manufacturer
Map osVersion to os version
Map cleansed_os to operating system
Map cleansed_model_sys_id to model ID
Map assigned_to to assigned to
Map chassisType to chassis type

Create OS Software Package

Select Add Conditional Class
Set the condition to operating system is not empty and the class to software
Select Set Up Mapping
Set the source native key source column to os_native_key
Set the key source column to os_native_key
Add the name and version columns via the add atrribute button
Map cleansed_os to name
Map osVersion to version

Create OS Instance

Select Add Conditional Class
Set the condition to operating system is not empty and the class to software instance
Select Set Up Mapping
Set the source native key source column to os_instance_native_key
Set the name source column to software_instance_name
Set the installed on to basic class and computer 1
Navigate back and on the software instance select Add associated class
Set the class to Software 1

If you receive an error whilst trying to associate the two classes, you can fix this by doing the following

Use the application navigator and search for Robust Transform Set Transformers
The name will be the name provided as part of basic details
Open the transform definition
Under the CMDB Integration Studio Entities related list locate Software Instance 1 and in the related for entity column set the value to Software 1
Navigate back to the IntegrationHub ETL record and you'll be able to toggle the software instance class to active
Mark the stage as complete

Add Relationships

Navigate back to the main ETL screen and select Add Relationships

Select add relationship
Set the computer 1 to the parent and network adaptor 1 as the child
Select Add and mark the stage as complete

Test & Rollback

One of the benefits of using the IntegrationHub is the ability to easily test and rollback the transforms, allowing for easy validation and changes without causing havoc to your sub-productions CMDB.

On the main ETL screen select Test and rollback integration results
Select run integration and await the results
Here is where you need to spend the time validating the created records are what you expect. If you've followed these instructions exactly you should see
- 5 mapped classes
- A number of relationships (it should be roughly 5 x number of records in the import set)
For errors and their fixes, check the docs IRE Error Messages

cynlink1 · ‎09-19-2022

Is it possible to obtain the device's IP address, installed software, and running processes from Azure/Microsoft Intune using this method?

Florian11 · ‎03-02-2023

Hi, thank you very much for your detailed instructions. I have also used it for myself and the CIs are created correctly. The problem I still have is that if there are duplicate devices with the same serial number in the Intune import set, several software instances are created for different versions of the operating system. One with the software version of the "old" device and one instance with the latest version, which is also in the OS version field of the CI.
How can it be achieved that in the case of duplicates, only the software instance of the most recent device is entered there?

Todor Kotsev · ‎05-18-2023

Hi,

I want to make sure the name is not updated and it synchs with the serial number. So if it finds the same serial number to update the CI. How can I do that

Thanks

jtshone · ‎12-16-2024

Did you find a solution as to how you define the coelesce on serial number for RTE...? I am finding it impossible to find any doc on this....

_ChrisHelming · ‎12-16-2024

@jtshone you need to set/alter the identification rules that are used. Open Class Manager, go to the class you want to modify, and select the "Identification Rule" section. They fire in order and stop when one matches. You can change or add rules to coalesce as you need.