AWS Discovery Issue with non-default regions (e.g., af-south-1)

srikanthnelapud · ‎05-18-2021

DESCRIPTION

Leveraging the AWS Master Member account discovery framework, Cloud Discovery triggered for the member account(s) whose, not just default but non-default regions are also activated, fails to fetch the Cloud resources i.e., Discovery fails with the error message - "AWS was not able to validate the provided access credentials" in the Cloud API Trail / Pattern Logs.

OBSERVATION

Firstly, when the discovery begins for such member accounts, the credentials i.e., Access Key and Secret Key is picked from the respective parent account i.e., AWS Master account, and a session token is generated. Secondly, Using the session token, obtained from the master account, is used to assumeRole over the member account and a session token is provided by the member account to use in the further API calls to retrieve the resources information.

We hit the global endpoint for the session token but not the region-specific which is really a problem because of few important lines across few articles -

New AWS Regions not enabled by default (for example, Hong Kong and Bahrain) use the updated AWS STS format. The global AWS STS endpoint (sts.amazonaws.com) issues tokens in the previous format by default. This error can occur if a session token is trying to use the previous format with an AWS Region not enabled by default."
"Version 1 tokens are valid only in AWS Regions that are available by default. These tokens do not work in manually enabled Regions, such as Asia Pacific (Hong Kong). Version 2 tokens are valid in all Regions. However, version 2 tokens include more characters and might affect systems where you temporarily store tokens."
"By default, AWS Security Token Service (STS) is available as a global service, and all STS requests go to a single endpoint at https://sts.amazonaws.com. AWS recommends using Regional STS endpoints to reduce latency, build in redundancy, and increase session token availability."
"If you make an STS call to the global endpoint, the resulting session tokens might be valid in some regions but not others. It depends on the version that is set in this operation. Version 1 tokens are valid only in AWS Regions that are available by default. These tokens do not work in manually enabled Regions, such as Asia Pacific (Hong Kong). Version 2 tokens are valid in all Regions. However, version 2 tokens are longer and might affect systems where you temporarily store tokens"

URL(S):

IAM Validate Access Credentials

Temporary Credentials, Enable Regions, STS Regions Manage Tokens

SetSecurityTokenServicePreferences API

WORKAROUND REFERENCE:

Section - "Regions and Endpoints in STS (Security Token Service)" of the below article - "Get the List of all AWS EC2 Instances with all their Private and Public IPs".
"An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials. This occurs because the IAM service where my IAM user exists (the master/management account) has the STS region compatibility of session tokens for the global endpoint policy set to only the legacy global regions, which makes the temporary credentials the STS service hands out useless against the af-south-1 region. Toggling the policy to “All AWS Regions” in the account where my IAM user exists (139769765351), and re-requesting the temporary credentials will work just fine for the af-south-1 region. You’ll also notice that once issued, the session token is longer – as the documentation states (here: “Tokens that are valid in all AWS Regions include more characters than tokens that are valid in Regions that are enabled by default“) – as it’s valid across all regions this time."

WORKAROUND:

Open the IAM console. In the navigation pane, choose Account settings.
If necessary, expand the Security Token Service (STS) section. In the first table next to the Global endpoint, the Region compatibility of session tokens column indicates Valid only in AWS Regions enabled by default. Choose Change.
In the Change region compatibility of session tokens for global endpoint dialog box, select Valid in all AWS Regions. Then choose Save changes.

SCREENSHOTS: Attached.