Q1. I want to give specific API permissions, but not generic IAM permissions like config:List*.
We are asking for read-only generic permissions to specific APIs as we can add more features and don't need to update the IAM policy every time we publish new versions. If you still want to give specific permissions, then here is the list of IAM and corresponding APIs:
| # | IAM Policy | API | Comments |
| 1 | organizations:DescribeOrganization | API Link | Required to get the Organization Account ID. This role is required where the ServiceNow user is being created. ## |
| 2 | organizations:ListAccounts | API Link | Required to get list of Account IDs. This role is required where the ServiceNow user is being created. ## |
| 3 | config:ListDiscoveredResources | API Link | Required at account level. Eventhough, if you are using Config Aggregator, we need this API access to get the list of deleted resources. This will keep AWS & CMDB in sync with all the latest and greatest information. |
| 4 | config:SelectAggregateResourceConfig | API Link | Required if you are using Config Aggregator. This role is required in the account where the Config Aggregator is created. |
| 5 | config:BatchGetAggregateResourceConfig | API Link | |
| 6 | config:SelectResourceConfig | API Link | Not required if you are using Config Aggregator |
| 7 | config:BatchGetResourceConfig | API Link | |
| 8 | ec2:DescribeRegions | API Link | Required only in the account where ServiceNow user created. |
| 9 | ec2:DescribeImages | API Link |
Required at each account level. This will get the AMI details with this permission. eg. ami-09d52511913f2928a (aws-parallelcluster-2.3.1-centos7-hvm-201904021550) |
| 10 |
ec2:DescribeInstances ec2:DescribeInstanceTypes |
API Link |
Required only in the account where ServiceNow user created. This will get the Hardware Type details with this permission. eg. t2.nano, t2.micro, t2.small etc. |
| 11 | ssm:GetInventory | API Link | Required at account region level to get the list of EC2 resources discovered by SSM Inventory. |
| 12 | ssm:ListInventoryEntries | API Link | Not required from version 1.5 onwards |
| 13 | ssm:DescribeInstanceInformation | API Link | |
| 14 | ssm:SendCommand | API Link | Required if you want to perform deep discovery to get EC2 Server attributes, TCP, Process information. |
| 15 | s3:GetObject | API Link | |
| 16 | s3:DeleteObject | API Link | |
| 17 | tag:GetResources | API Link | Required at account level. Some Resources like ELB, DynamoDB, Lambda doesn't return Tags with Config API. This API is being called to get the tags of the resources. You can ignore if you don't need tags. |
| 18 |
iam:CreateAccessKey
|
API Link | Creates a new AWS secret access key and corresponding AWS access key ID for the specified user. |
| 19 |
iam:DeleteAccessKey
|
API Link | Deletes the access key pair associated with the specified IAM user. |
## - Some companies place organization AWS service control policies to restrict member accounts to make specific API calls. You need to talk to your corporate security operations team to whitelist (Allow) the member accounts.
Please run the CLI commands listed below to verify that the ServiceNow user has the appropriate permissions and that you are receiving a response. We should be able to obtain the data once this step is completed.
aws organizations describe-organization
aws organizations list-accounts
Q2. I've added all the required permissions, but I'm still getting a HTTP 400 response.
Please check with your AWS admin if there is any corporate IAM policy denying you access.
Q3. I tried to re-import all of the data, but I am not getting it.
SG-AWS will update the 'Last run datetime' column after every import schedule. This can be found in the Data Sources table as shown below. If you want to perform a fresh pull, you need to clear the 'Last run datetime' column in all of the rows. The application assumes this as bootstrap load and will pull in all of the data.
Q4. Discovery populates Linux/Windows server details in Linux (cmdb_ci_linux_server)/ Windows (cmdb_ci_win_server) CI tables. Why does the SG-AWS populate Linux/Windows server details in Server (cmdb_ci_server) class?
We don't populate Linux/Windows server classes purposefully and map only to the server class. There are specific discovery patterns that will detect specialized types of servers that refers to Linux and Windows servers in the server hierarchy. So we map to the parent class and let discovery re-classify the records to the specific server classes.
As part of SGC-AWS 2.0 release, we will start classifying Server classes and populate in respective CI types. Please refer to Section 5 in the article https://www.servicenow.com/community/cmdb-articles/sgc-aws-release-2-0-features/ta-p/2511663
Q5. Do I need to execute AWS-SystemsManager-AutomationAdministrationRole.yml & AWS-SystemsManager-AutomationExecutionRole.yml?
SSM Inventory needs to be set up in all of the account regions you wish to discover. You can set up SSM Inventory by logging into each account region and set up with SSM Automation scripts. For example, if you have one hundred accounts and an average of four regions, you need to repeat these steps 400 times, which can take days and weeks. To simplify the setup, these IAM role scripts are executed by your AWS Admin to facilitate setting up SSM Inventory to all the member account regions. For more details, refer to the AWS documentation.
Q6. I am using Config Aggregator, why do I need access to the Account level ListDiscoveredResources Config Access?
AWS API captures the deleted resources in to its config database. Currently it captures only at an account region level and not at Aggregator level. There is a feature request being raised to get this detail at aggregator level. Until this feature is fulfilled, we need to access at an account region level to update the deleted resources.
ListDiscoveredResources gives the list of resources in each account region level. It has a flag 'includeDeletedResources' which is set 'true' if the resource is deleted from AWS.
SelectAggregateResourceConfig which is used to get the list of resources from AWS Config API, has the following nine attributes and doesn't have deleted resource details. Hence, we hop into an account level with ListDiscoveredResources API to get the deleted resource details.
- Account ID
- Resource ID
- AWS Region
- Availability Zone
- Resource Type
- Version
- ARN
- Configuration Capture Time
- Resource Creation Time
Q7. What are the APIs being used in SGC-AWS?
| CMDB | CI Class | AWS API Source | Resource Type / Query Param Action | API | API Reference |
| Organization | cmdb_ci_cloud_org | Organizations API | Query Param Action: Action=DescribeOrganization | https://organizations.us-east-1.amazonaws.com | DescribeOrganization |
| Cloud Service Account | cmdb_ci_cloud_service_account | Organizations | Query Param Action: Action=ListAccounts | https://organizations.us-east-1.amazonaws.com | ListAccounts |
| AWS Datacenter | cmdb_ci_logical_datacenter | Organizations, EC2 Regions | |||
| Availability Zone | cmdb_ci_availability_zone | Most APIs | |||
| Hardware Type | cmdb_ci_compute_template | EC2 API | DescribeInstanceTypes | https://ec2.<aws_region>.amazonaws.com?Action=DescribeInstances&Version=2016-11-15&InstanceId.1=i-0fd44c92ffd01dc68 | DescribeInstances |
| Image-Private | cmdb_ci_os_template | EC2 API | DescribeImages | https://ec2.<aws_region>.amazonaws.com?Action=DescribeImages&Version=2016-11-15&Owner.1=self&Filter.1.Name=state&Filter.1.Value=available&Filter.2.Name=is-public&Filter.2.Value=false | DescribeImages |
| Image-Id | cmdb_ci_os_template | EC2 API | DescribeImages | ||
| Cloud Network | cmdb_ci_network | AWS Config API | AWS::EC2::VPC | https://config.<aws_region>.amazonaws.com Action: BatchGetAggregateResourceConfig Action: BatchGetResourceConfig Action: ListDiscoveredResources Action: SelectResourceConfig Action: SelectAggregateResourceConfig |
ListDiscoveredResources |
| Cloud Subnet | cmdb_ci_cloud_subnet | AWS Config API | AWS::EC2::Subnet | ||
| Cloud Mgmt Network Interface | cmdb_ci_nic | AWS Config API | AWS::EC2::NetworkInterface | BatchGetAggregateResourceConfig | |
| Compute Security Group | cmdb_ci_compute_security_group | AWS Config API | AWS::EC2::SecurityGroup | ||
| Storage Volume | cmdb_ci_storage_volume | AWS Config API | AWS::EC2::Volume | SelectAggregateResourceConfig | |
| Cloud Load Balancer- Classic | cmdb_ci_cloud_load_balancer | AWS Config API | AWS::ElasticLoadBalancing::LoadBalancer | ||
| Cloud Load Balancer - V2 | cmdb_ci_cloud_load_balancer | AWS Config API | AWS::ElasticLoadBalancingV2::LoadBalancer | ||
| Cloud Gateway | cmdb_ci_cloud_gateway | AWS Config API | AWS::ApiGatewayV2::Api | ||
| Virtual Machine Instance | cmdb_ci_vm_instance | AWS Config API | AWS::EC2::Instance | ||
| Server | cmdb_ci_server | AWS Config API | AWS::EC2::Instance | ||
| Cloud Database | cmdb_ci_cloud_database | AWS Config API | AWS::RDS::DBInstance | ||
| Cloud Function | cmdb_ci_cloud_function | AWS Config API | AWS::Lambda::Function | ||
| Cloud Object Storage | cmdb_ci_cloud_object_storage | AWS Config API | AWS::S3::Bucket | ||
| DynamoDB Table | cmdb_ci_dynamodb_table | AWS Config API | AWS::DynamoDB::Table | ||
| Software | cmdb_ci_spkg | AWS Config API | AWS::SSM::ManagedInstanceInventory | ||
| SendCommand (TCP, Running Process) | cmdb_running_process | SSM API | X-Amz-Target: AmazonSSM.SendCommand | https://ssm.<aws_region>.amazonaws.com | https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_SendCommand.html |
| cmdb_tcp | SSM API | X-Amz-Target: AmazonSSM.ListCommandInvocations | ListCommandInvocations | ||
| S3 Get | https://s3.<aws_region>.amazonaws.com (HTTP: GET) | GetObject | |||
| S3 Delete | https://s3.<aws_region>.amazonaws.com (HTTP: DELETE) | DeleteObject | |||
| Tags | X-Amz-Target: ResourceGroupsTaggingAPI_20170126.GetResources | https://tagging.<aws_region>.amazonaws.com | https://docs.aws.amazon.com/resourcegroupstagging/latest/APIReference/overview.html |
Config API calls are made depending on the configuration you have done in AWS side. Here is the flow diagram which specific API is being called.
8. Deleted Resources are not reflecting in CMDB.
SGC-AWS marks Status field as 'Retired' when the resource is terminated in AWS. With daily schedule it gets this information.
For some reason, if the schedule is not running daily, it might cause disconnect with AWS. ListDiscovered resources API returns all the resources deleted based on the retention period. Hence we will filter the records to accept the resources which got deleted last 3 days using the flag in sys_parameter - sn_aws_integ.delete_resource_timeline =3. To fill the gap, you can change this value to 10/30 or some number you want to fix the inconsistent state issue and then revert back to 3.
However, there are other 2 states - Operational status and State fields are not updated during deletion strategy. Hence you need to refer to 'Status' field.
Related Articles:
Service Graph Connector for AWS - Introduction
Service Graph Connector for AWS - Functional Spec and CI Details
Service Graph Connector for AWS - Diagnostic Tool
