Service Graph Connector for AWS - Functional Spec and CI

Murali Reddy1 · ‎03-22-2022

Functional Requirements

Functional requirements are broadly classified into two categories: Hardware and Software. To pull CI information from AWS, follow the three data pull strategies below:

Baseline

This is fresh import of data from AWS into the CMDB CI Database.
After importing, we will record the time of import, this will be the input for the incremental pull.

Incremental Update/Delta:

Delta or Incremental update will pull the resource data which was updated or created after the last pull.

Deletion

We will identify the resource as deleted and perform the operation based on the hardware or software type.

Discovered Hardware CIs

#	Class Name	CI Name	API Source
1	Cloud Organizations	cmdb_ci_cloud_org	Organizations
2	Cloud Service Account	cmdb_ci_cloud_service_account	Organizations
3	Logical Datacenter	cmdb_ci_logical_datacenter	EC2
4	Availability Zone	cmdb_ci_availability_zone	AWS Config
5	Cloud Network	cmdb_ci_network	AWS Config
6	Cloud Subnet	cmdb_ci_cloud_subnet	AWS Config
7	Cloud Mgmt Network Interface	cmdb_ci_nic	AWS Config
8	Hardware Type	cmdb_ci_compute_template	AWS Config
9	Compute Security Group	cmdb_ci_compute_security_group	AWS Config
10	Image	cmdb_ci_os_template	AWS Config
11	Storage Volume	cmdb_ci_storage_volume	AWS Config
12	Cloud Load Balancer	cmdb_ci_cloud_load_balancer	AWS Config
13	Cloud Gateway	cmdb_ci_cloud_gateway	AWS Config
14	Virtual Machine Instance	cmdb_ci_vm_instance	AWS Config
15	Server	cmdb_ci_server cmdb_ci_linux_server cmdb_ci_win_server	AWS Config
16	Cloud DataBase	cmdb_ci_cloud_database	AWS Config
17	Cloud Function	cmdb_ci_cloud_function	AWS Config
18	Cloud Object Storage	cmdb_ci_cloud_object_storage	AWS Config
19	DynamoDB Table	cmdb_ci_dynamodb_table	AWS Config
20	IP Address	cmdb_ci_ip_address	AWS Config
21	Network Adapter	cmdb_ci_network_adapter	AWS Config
22	TCP	cmdb_tcp	SSM Send Command & ADM process
23	Running Process	cmdb_running_process	SSM Send Command & ADM process
24	VNIC Endpoint	cmdb_ci_endpoint_vnic	AWS Config
25	Storage Mapping	cmdb_ci_storage_mapping	AWS Config
26	Block Endpoint	cmdb_ci_endpoint_block	AWS Config
27	Key Value	cmdb_key_value	AWS Config
28	Application	cmdb_ci_appl	SSM Send Command & ADM process

Discovered Software CIs


SAM Plugin	Class Name	CI Name	API Source
Installed	Software Install	cmdb_sam_sw_install	SSM & AWS Config
Not Installed	Software Package	cmdb_ci_spkg
Not Installed	Software Instance	cmdb_software_instance

Consolidated view of CIs

Detailed CI Information

Cloud Organizations [cmdb_ci_cloud_org]

Name
Data Source	SG-AWS

Cloud Service Account [cmdb_ci_cloud_service_account]

Account Id
Name
Datacenter Type
Parent Account
Is master Account
Data Source	SG-AWS

Logical Datacenter [cmdb_ci_logical_datacenter]

Name
Data Source	SG-AWS

Availability Zone [cmdb_ci_availability_zone]

Name
Data Source	SG-AWS

Cloud Network [cmdb_ci_network]

Name
Object ID
Status	Installed
Data Source	SG-AWS
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters

Cloud Subnet [cmdb_ci_cloud_subnet]

Name
State	Available
Object ID
CIDR
Status	Installed
Data Source	SG-AWS
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters Key-Value

Cloud Mgmt Network Interface [cmdb_ci_nic]

Name
Netmask
Status	Installed
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters Key-Value
Data Source	SG-AWS

Hardware Type [cmdb_ci_compute_template]

Name
vCPUs
Memory MB
Logical Storage GB
Data Source	SG-AWS

Compute Security Group [cmdb_ci_compute_security_group]

Name
ObjectID
State	Available
Related Items	Cloud Networks Availability Zones Cloud Service Accounts Logical Datacenters Key-Value
Data Source	SG-AWS

Image [cmdb_ci_os_template]

Name
ObjectID
Guest OS
Root Device Type
Image Type
Image Source
Data Source	SG-AWS

Storage Volume [cmdb_ci_storage_volume]

Name
ObjectID
State	Available/In Use/Terminated/Leased
Size
Storage Type
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters Key-Value
Data Source	SG-AWS

Cloud Load Balancer [cmdb_ci_cloud_load_balancer]

Name
ObjectID
Related Items	Cloud Load Balancers Availability Zones Cloud Service Accounts Logical Datacenters Compute Security groups Virtual Machine Instances Cloud Object Storages Images Hardware Types Cloud Gateways Storage Volume Snapshots Storage Volumes Cloud Networks Cloud Databases DynamoDB Tables Cloud Mgmt Network Interfaces Key-Value
Data Source	SG-AWS

Cloud Gateway [cmdb_ci_cloud_gateway]

Name

Related Articles:

Service Graph Connector for AWS - Introduction

Service Graph Connector for AWS - Diagnostic Tool

conmic · ‎04-04-2022

Hi Murali,

First of all, this is all very awesome! However, I'm confused now between the AWS Cloud Discovery capability and this Service Graph Connector.

I understand that there is a difference when it comes to licensing (one is ITOM Visibility, the other Service Graph), but now we basically have two overlapping products doing the same thing, or to some extend not?

What I am now looking for is: What is the difference between these two products and which one should I chose first if the license discussion is not really of concern?

Thanks,
Best regards,
Michel

vaibhavbhatnaga · ‎04-04-2022

Hi Michel,

Cloud Discovery -

API-based Discovery
Cloud resource Inventory and Tags (metadata)

And to do Deep dive Discovery – Server, OS, software, TCP connections, processes & relationships, IP discovery needs to be set up. And MID is recovered for this.

However, for SG-AWS no MID is required and it does –

API-based Discovery
Cloud Resource Inventory, Tags and Relationships
Deep dive discovery – Server, OS, Software, Processes

SG-AWS is an option where we need deep dive Discovery but credentials and/or MID Servers are an issue.

Thanks

conmic · ‎04-07-2022

Hi,

Thanks you so much, very helpful!

I also see that you went out of your way to create a new article on this subject, much appreciated!

Best regards,
Michel

Pranav Patil · ‎08-24-2022

Hi Conmic,

If License is not an issue then Discovery is better choice over the graph connector. As per my understanding Graph connector will be used if there is no discovery (License) available.

Both of the applications have their own advantages

Regards,
Pranav Patil

David Clarkson1 · ‎09-15-2022

From the latest I've heard, the AWS Service Graph connector is only available bundled with other SKUs, like Discovery or HAM Pro. Per ServiceNow's guidance, it seems the AWS SG Connector is the recommended approach even if the client has Discovery, because of the architectural and performance advantages.

Charles Lowe · ‎12-09-2022

Can this connector replace Cloud Discovery if we are using ServiceNow Service Mapping?

Murali Reddy1 · ‎12-09-2022

SGC-AWS is not a replacement to Cloud Discovery. Its an alternative mechanism for customers who doesn't want to use MID layer. SGC architecture has IntegrationHub ETL for you to customize the mapping if required https://www.servicenow.com/products/service-graph-connectors.html, https://docs.servicenow.com/bundle/rome-servicenow-platform/page/product/configuration-management/co... Let me know if this answers your question

Aarti6 · ‎12-24-2022

@Murali Reddy1 Hi Murli,

Thanks for the detailed article.

I was referring the steps to install SGC for AWS, in the first step of Guided setup it is asked to run few scripts at AWS side. "Download the scripts" and run in Cloud Formation module in AWS.

Can you please give us more idea on what are these scripts exactly?

Murali Reddy1 · ‎01-05-2023

Hello @Aarti6, the download scripts are part of the SGC-AWS plugin's guided setup. These scripts basically setup the environment like setting up AWS Config, IAM roles for Servicenow user, setting up SSM Inventory, etc. These setup is required for us to securely integrate with AWS environment.

Screen Shot 2023-01-05 at 11.49.04 AM.png

The setup instructions for each script is described in the guided setup "AWS Setup Instructions"

Screen Shot 2023-01-05 at 11.51.52 AM.png

Aarti Aushal · ‎01-12-2023

Thanks Murali

Rawel · ‎04-03-2023

This connector is creating some issues for us.

SG-AWS is creating a VM instance which is related to a Server CI (i.e. cmdb_ci_server and not Windows/Linux). And it is using the field privatednsname for setting the name of the Server CI (which as per AWS standard is ip-x-x-x-x). We have also discovered the same server CI via IP based discovery and have a separate Windows/Linux CI in the system.

So now we have a VM instance, a Server CI and a WIndows/Linux server.

Murali Reddy1 · ‎04-19-2023

As part of upcoming 2.0 release in May 2023, we will be populating in Windows/Linux server. Please see

https://www.servicenow.com/community/cmdb-articles/sgc-aws-release-2-0-features/ta-p/2511663.

However, in case of name, we have to be in sync with discovery product (which populates name as ip-x-x-x-x) to avoid duplicate entries.

Rawel · ‎04-19-2023

Thanks for the update Murali.

But why is the connector using privatednsname for populating Name of the Server CI. This will just now create a new Windows/Linux CI with name as ip-x-x-x-x. (I confirmed with our cloud team and this is how AWS populates privatednsname and there is nothing the cloud team can do)

This can be easily avoided by updating the mapping to use ec2name field which is already happening for VM instances.

Murali Reddy1 · ‎04-21-2023

@Rawel PrivateDNS name is being used in Discovery product and some clients are using both SGC & Discovery. Having a different name will create duplicated and we have to be in sync with discovery product. If you want, you can customize in ETL to have ec2Name or resource id or any other custom name you want to define. . Hope this helps.

jnipala · ‎05-04-2023

Cloud Load Balancer and Cloud Gateway in picture is under Datacenter but NO relation to VPC.

Is this correct since I think those are under VPC so there should be dependency to VPC instead datacenter or then both ?

Kieren · ‎07-24-2023

@Murali Reddy1
We are finding the SG is causing a few duplicates due to naming as well.

IP Based discovery is setting correlation_id and object_id to the VM's Instance Id. To me this appears to be a more reliable way to match the host. Do you see any issues in that?

With a reconciliation rule we can then prevent the SG from setting ip-x-x-x-x naming for hosts that were discovered by IP.

philj-ensono · ‎08-30-2023

We are actively using the AWS SG connector. Could you add support for more of the resource types that AWS config can provide please to start to fill some of the gaps in the current discovery ? ( e.g. Cloudfront distributions, EFS file systems, Redshift clusters, ECR repos, ACM certificates, Autoscaling groups and launch configs, Route 53 zones, SNS topics, SQS queues.... ). Thankyou.

James99 · ‎02-21-2024

@Murali Reddy1 regarding using the ETL to re-map the 'Name' field to use the ec2name, we've tried this on our instance, but it's only updating a small number of servers.

Is there a setting which is only detecting delta changes in the environment for recognizing the updates made in ETL? As this was the case with classifying servers into the Windows and Linux classes as found on CS6979561.

Kieren · ‎02-27-2024

@James99 The AWS SG does run as delta, it will only get data back from AWS for EC2s that were changed since the last run.

The delta time is stored in the Last run Datetime field on the datasource

Anna Lee1 · ‎03-31-2024

Hi @Murali Reddy1,

Thank you for your beneficial information.

From your information, SGC-AWS can deep dive discovery - for example, Server, OS, Software, and Processes.

If I need to discover WEB and WAS CIs on EC2, does SGC-AWS support this?

Regards,

Anna

Bhims · ‎10-21-2024

Hi @Murali Reddy1

As a part of AWS ECS, what all services are being pulled by SG-AWS, can you share some information around it?

Regards

Bhim

Name
Operating System
Serial Number
DNS Domain
Disk Space (GB)
RAM (MB)
CPU Speed(MHz)
CPU count
CPU core count
Manufacturer
Model ID
CPU Manufacturer
CPU Type
CPU Speed (MHz)
CPU Core count
Operating System
OS Version
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters Images Hardware Types Webservers SQL Instances Block Endpoints Virtual Machines Key-Value
Data Source	SG-AWS

PID
Name
Command
Parameters
Key parameters
PPID
Parent
Connecting to
Listening on
Computer
Absent	false/true
Data Source	SG-AWS

Name
ObjectID
Disks
Network adapters
IP Address
VM Instance ID
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters Images Hardware Types Servers Key-Value Webservers SQL Instances
Data Source	SG-AWS

Name
TCP port
Fully qualified domain name
Version
Type
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters Key-Value
Data Source	SG-AWS

Name
Language
Version
State
Function Last Modified
Status
Code Size
CodeSha256
Related Items	Availability Zones Cloud Service Accounts Logical Datacenters Key-Value
Data Source	SG-AWS

Name
MAC Address
IP Address
Configuration Item
Data Source	SG-AWS