Service Graph Connector for AWS - Diagnostic Tool & Troubleshooting Issues

[Murali Reddy1]

Service Graph Connector for AWS (SG-AWS) provides CloudFormation (CFT) scripts for appropriate setup in an AWS environment. There are tens or hundreds of administrators managing AWS accounts. The scripts are designed to be executed from an organization/master account to all member accounts. Some companies may decide to set up their environment using their own scripts or be given to individual administrators. To successfully integrate all of the accounts with SG-AWS, we need to have the setup done correctly. On the client side, someone managing the ServiceNow instance and AWS admins may not be an expert with a ServiceNow instance. To check if all the setup is done correctly, the diagnostic tool is useful along with the SG-AWS guided setup.

As described in the Service Graph Connector for AWS - Introduction article, SG-AWS depends on AWS Config, SSM components, and IAM permissions. This tool helps in checking the ServiceNow user has the appropriate IAM permissions in all accounts and APIs used. 

It is advised to execute the Diagnostic Tool after setting up the AWS Setup and adding the details in the guided setup. You can execute the Diagnostic Tool as many times as needed. The tool helps you, the support team, and the development team to identify the issues and advise the remediation steps. If there is any issue in the setup, the tool points you to the right documentation to follow and fix the issues. 

Where can I find this diagnostic tool?

The diagnostic tool is available as part of Service Graph Connector for AWS plugin in the guided setup.

Triggering Diagnostic tool:

Once you click on 'Configure' in the guided setup, you will be directed to the Diagnostic Tool page as shown below:

Once you click on 'Run Diagnostic Test' button, it triggers a backend asynchronous job to perform the diagnostic process. The button will be disabled until the process is completed. Depending on number of accounts, it may take a few seconds to a few minutes. 

Visibility of diagnostic tests:

As you can run as many tests you need, the tool is designed to show the last 15 tests in descending order. You can pick the required tests from the 'Select Diagnostic' drop-down menu. 

The Diagnostic tool consists of three components:

1. Diagnostic Summary - Summary of diagnostic tool.
2. Diagnostic Summary Notes - Running notes of diagnostic tool execution.
3. Diagnostic Summary Detail - HTTP status of each API call in every account in a tabular format. 

Diagnostic Summary

Diagnostic summary gives the summary of the test:

• Unique ID of the diagnostic test.
• Number of accounts in the organization.
• Number of regions configured.
• Summary of API calls count gives number of successful call. The detail is described in summary detail.
• Some API calls like Image API, Hardware Type API etc which are common across accounts have HTTP status.
• Diagnostic state shows the state of the diagnostic test. 
• InProgress - Diagnostic Test in progress:
• • Failed - State will be marked failed if primary API calls like AWS Account access, Region API, Image API, Hardware Type API calls failed. When any of these API calls fails, the SG-AWS cannot pull the other API details successfully.  
  • Completed - Primary API calls are passed and completed executing account level API tests. The status will be marked as complete even if some account level API calls fails. You need to look in the Diagnostic Summary details to see which API has the HTTP status 400/403. 

Below images shows the successful completion of the diagnostic test. 

Diagnostic Summary Notes

• Summary notes will give details of running notes of the diagnostic tool.

Feature	Snapshot	Description
Property snapshot		Diagnostic tool will capture the properties set for this application for troubleshooting purposes.
Central Aggregator		This shows Central Aggregator is setup and the application will use aggregator for accounts.
Management Account		This means customer has created ServiceNow user in Management account. If you have created ServiceNow user in member (designated) account, then you need to set this value.
Primary API Calls Test		To get complete CIs, you need to have access to list account(s), get AWS regions. If this call fails, then the diagnostic test will report as FAILED.
Key APIs tests		The tool will log critical API tests and Marks as Passed / Failed for the you and support team to troubleshoot the issue.
Region Level API Calls log		Some API calls that needs to be tested at region level, the tool logs as FAILED for you to troubleshoot the issue. In this case, for the given account, the API call failed in us-east-1 and us-east-2 region. You need to check if the SG-AWS-RunShellScript SSM document exists in the region and have proper IAM access.
Test Completed		This message shows, the tests are completed.

 Diagnostic Summary Detail

The Diagnostic Summary detail gives the tabular format view of the API calls made for each account with a HTTP Status code. If you get a HTTP 2xx code response, it means SG-AWS was able to make a successful API call and can process the data. If you see a HTTP 4xx code, then you need to look into the account with the proper IAM privileges. 

If you have selected Central Aggregator, then Config Batch API & Config Select API calls will not be made and they will be empty. 

Troubleshooting

Error	Error Description
	This error comes up when you entered wrong credentials or not having proper IAM access in the account. Please follow the setup instructions

Q1 : SSM / S3 API not giving desired counts, why is this behavior?

As part of guided setup, you are given an option to enter the regions you are interested to import the CI. As shown in the below, the user has entered 4 regions (us-east-1,us-east-2,us-west-1,us-west-2). SG-AWS will import the data only from these 4 regions. If this field is set empty, SG-AWS will import from all 21 regions. In case of SSM, we need to deploy the SSM Documents (SG-AWS-RunShellScript, SG-AWS-RunPowerShellScript), in all/preferred account regions. 

The Diagnostic Tool is set up to test the IAM permissions set for the ServiceNow user in all the accounts. Hence, if the region value is set to empty, it will check one region (us-east-1). Why is this the behavior? This is done to optimize the performance. If you have thousands of accounts and test them in all 21 regions, it will take a very long time for all the APIs to be tested. Hence, as we are testing for the IAM role access which is at global level, it's enough to test with one region. 

Guided Setup Regions set to us-east-1,us-east-2,us-west-1,us-west-2	No region preference set in Guided Setup
There are 5 accounts and 4 regions and the diagnostic tool will check in 5 * 4 = 20 account regions for SSM Document Access. In this scenario, we deployed SSM document in 15 regions and hence you see 15/15 count. It means, SSM document is deployed in 15 regions and we were successfully able to execute and get desired results from 15 account regions.	In this scenario, as the regions are not set in guides setup, the application will test in default (us-east-1) region in each account and report the count. Hence, as we have 5 accounts and SSM/S3 APIs will be tested in us-east-1 region and reported the count 5/5.

Related Articles: 

Service Graph Connector for AWS - Introduction

Service Graph Connector for AWS - Functional Spec and CI Details

ServiceGraph AWS Connector - Using MID Server

Cloud Discovery and SG-AWS

Service Graph Connector for AWS - FAQ

Service Graph Connector for AWS - SSM Documents

[nr_smartinez]

Hello  Murali,

Hoping you can help. I have set this up in my PDI using our demo AWS as an org user. Each time, using the diagnostic I get an error stating "Cannot read property "keys" from undefined"? 

[Murali Reddy1]

Hello Steven, Can you raise a case task to look in to the issue. The tool covers lots of use cases and it was developed and tested most common environment. Once we see the specific issue and we can fix the tool with this specific failure. 

[nr_smartinez]

Just did this morning! Thanks!

[Pranav Patil]

Hi @Murali Reddy1 

I am working to pull data from 2 AWS organizations. Can we setup multiple / one plus organizations in the property ? 

Regards,
Pranav Patil 

[Murali Reddy1]

Hi @Pranav Patil,

Multi-Org will be supported from upcoming release SGC-AWS 2.0 from May 1st week. 

Thanks,

Murali

[Denisa Mary]

Hi @Murali Reddy1 ,

We have issues with the Diagnostic Tool, the status shows "Diagnostic Test in progress" and the 'Select Diagnostic' drop-down menu not showing the ID's.  This is the status for past 4 days and we do not find any long running job and verified the sys_progress_worker table no job running related to the diagnostic tool in aws.

Kindly let us know how to to fix this issue.

Thanks,

Denisa

[Murali Reddy1]

Hello @Denisa Mary, Normally it should be done in < 30 mins for 115 accounts. We have to investigate deeper why it got struck in middle. As you say, there is no active process running, probably someone might have killed the process. 

The tool prevent you to run multiple tests and hence the buttons are frozen. To overcome the issue you can go to sn_aws_integ_sg_aws_diagnostic_summary table and change the status to COMPLETED and start the diagnostic test again. Hope this helps. 

If you still see issues repeating, pls raise a case task, our team can help. 

Thanks,

Murali

[Abdul Parveez]

@Murali Reddy1  - Hi Murali, we are getting below error while setting up SGC-AWS. there is not many detail in the error.  AWS  team can see the attached event when we run diagnostic tool.

[Pranav Patil]

@Abdul Parveez 

We faced the same issue however I ran the scheduled import job once and it fetched data from AWS end. 

@Murali Reddy1  Can you please share why this error appears of "Organization account access details API test failed" as we have double checked all configurations from both service now and AWS end are appropriate. 

Thanks 

[Abdul Parveez]

tried the import schedule, but it is not getting any data

[adinad]

Hello,

I am having the same error as @Abdul Parveez  .

Did you find a solution? Not sure what else to check:

[Abdul Parveez]

@adinad  - the error you are getting is due to security IAM roles, ask your cloud IAM team to verify the latest scripts are executed and proper permissions are given to ServiceNow user. My original error was fixed by upgrading the plugin. I did have same error that you have shared but tat was fixed by our cloud IAM(Security) team.

[IlyaN]

@Abdul Parveez , @adinad
Hello, 

I have the same error and I double-checked that IAM is set up to allow org description just as in the download scripts. Also made sure that the plugin was up to date and ran an import job from Org but no results. 

Organization Account Details API test - FAILED with HTTP Status Code 400

12	Organization Account Access API [Designated. Account] FAILED. Please check credentials are correct.
13	Organization Account Details API test - FAILED. Check if the ServiceNow user has IAM roles (organizations:Describe*, sts:AssumeRole) set up properly. Refer to SG AWS Setup Instructions > ServiceNow user Setup.

May I please get some guidance on this? 

Thanks in advance!

[KrushnakumaT]

How to overcome below errors in SGC - AWS