Unable to Discover with Terraform Cloud

[Brent Langdon]

I am quite new to ServiceNow.  I am working with a developer instance to develop a proof of concept to integrate ServiceNow with Terraform.  I got it working with a "Terraform Environment" (a Linux vm with Terraform installed).  I could not get Discovery to work with Terraform Cloud.  I am looking for any tips on what I might be doing wrong.

I created a Config Provider with the following settings:

• Provider: Terraform Enterprise
• Server Type: Cloud
• Organization: my organization in Terraform Cloud
• URL: https://app.terraform.io/api/v2/
• Credentials: (created using API Token from Terraform Cloud)

Based on some trial and error, I think the URL is correct.  I have tried using an API Token for the User, Group and Organization without any luck.

When I run this, I see some lines in my MID Server log that look valid and don't report any errors.

I don't understand what "Ensure that you create a credential alias with Type > Connection." from this page:
https://docs.servicenow.com/bundle/orlando-it-operations-management/page/product/cloud-management-v2-setup/task/terraform-enterprise-config-provider.html

Not sure if that is supposed to say "Type > Credential".  I added an alias of type Credential, but I don't really know what that does.  Adding an Alias of Type "Connection and Credential" generates a validation message: "Alias X is not of type Credential"

I tried upgrading my dev instance to Orlando to see if that would make a difference, but it did not.

I see the following entries in my ServiceNow System Log each time I run Discovery (messages at the bottle are generated first).

==========
Failure while processing chunk : ChunkedStepResult{correlationId='68afdebfdbcf00102655d426ca961901', error='
Outbound REST call to get TFE VCS failed StatusCode:401, ErrorCode:3, Message:Method failed: (/api/v2/organizations/ngc-cssg/oauth-clients) with code: 401 - Invalid username/password combo', output='[]', chunkNumber=1, totalChunks=1, isRouteComplete=true, exception=null}:
==========
Failure to handle chunked result: com.snc.cloud.mgmt.modules.svccatalog.orchestration.BPOException:
Outbound REST call to get TFE VCS failed StatusCode:401, ErrorCode:3, Message:Method failed: (/api/v2/organizations/ngc-cssg/oauth-clients) with code: 401 - Invalid username/password combo:
==========
Failed to parse the output of new CMPCIRelationshipUtil().getLdcAndServiceAccount('<param value removed>'), exception: : com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
==========
Failed to parse the output of new CMPCIRelationshipUtil().getLdcAndServiceAccount('<param value removed>'), exception: : com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
==========

I am able to call /api/v2/organizations/ngc-cssg/oauth-clients using curl (from a command prompt on my Linux based MID Server) and passing in any of the 3 API Tokens (User, Group or Org) following the Terraform Cloud API. (example here: https://www.terraform.io/docs/cloud/api/oauth-clients.html)

Any tips appreciated!  Thanks!!

[Ashok Madhavan1]

Yes, you rerun the discovery and it would populate CMDB.

For newer projects you would create newer catalog items.

For existing catalog items you could add a new version of the template.

[Brent Langdon]

Regarding this section of the document:

Any repo under any of these VCS providers can be used by Cloud Management. For this we will need credential information about the VCS system so that we can read the Terraform files for the ingestion process which will come later.  This is something which is very critical for catalog item creation. You would need to create the Github credential record for the VCS system(s).  As usual you create the API key credential record and associate it to the VCS records. 

Create a API Key Credential by going to the credentials menu. The click New. Then choose an API Key Credential. 

Key for VCS system should be like this token<SPACE>< GENERATED-KEY >. (token fabcdefgh123.....…)

What token is being used here?  The same API Token generated from Terraform Cloud that were used on the initial Config Provider?

I have Terraform Cloud hooked up to a public Github repository.  I have setup OAuth, but don't see where that would fit in.  I tried creating a Personal Access Token in Github, but that did not seem to work.  I looked at creating an SSH Key to connect to Github, but that does not seem to be what is needed here.

[Ashok Madhavan1]

we use personal access token for github. You can get there via your Settings and then Developer Settings.

[Brent Langdon]

Thanks for the help, but that is just not working for me.  I am not positive what the issue is.

I can run the following from the command line from my MID server and that works, so the Personal Access Token that I am using appears valid:

export OAUTHTOKEN=c0dd....
curl -H "Authorization: token $OAUTHTOKEN" https://api.github.com/user/repos

When I click the "Find Branches and Repositories" link it spins for a while and I get a message:

Failed to retrieve repository and file details, please check the orchestration logs and flow logs for more details.

Looking at the orchestration logs, I see a handful of entries, with 2 Error records.  The one with a stacktrace says the following.  Note that I see the same results if I enter a bogus token, so the issue might be with something else.

Missing VCS system information in inputdetails: Input validation failed com.snc.cloud.mgmt.modules.svccatalog.orchestration.BPOException:
Missing VCS system information in inputdetails: Input validation failed
    at com.snc.cloud.mgmt.modules.svccatalog.orchestration.result.StepResultHandler.checkFlowReturnStatus(StepResultHandler.java:287)
    at com.snc.cloud.mgmt.modules.svccatalog.orchestration.result.StepResultHandler.validateResult(StepResultHandler.java:169)
    at com.snc.cloud.mgmt.modules.svccatalog.orchestration.result.StepResultHandler.process(StepResultHandler.java:92)
    at com.snc.cloud.mgmt.modules.svccatalog.service.impl.BlueprintOrchestratorImpl.handleChunkedResult(BlueprintOrchestratorImpl.java:82)
    at com.snc.cloud.mgmt.modules.svccatalog.service.impl.CloudOrchestrationServiceImpl.handleChunkCompletion(CloudOrchestrationServiceImpl.java:312)
    at com.snc.cloud.mgmt.modules.svccatalog.scriptinterface.BPOrchestratorServiceScript.jsFunction_handleFlowResult(BPOrchestratorServiceScript.java:158)
    at sun.reflect.GeneratedMethodAccessor2299.invoke(Unknown Source)

[Brent Langdon]

Just to follow up on this...I was able to get the basics of the integration working.  The two additional issues I had were:

• I am not sure where the "Name" field on the Terraform Enterprise VCS record is supposed to come from or what it represents.  I had noticed that the record created when I ran the initial "Discover Now" had a blank Name where the examples in the linked article had names.  When I dug down into the scripts, there was validation on the name that caused the "Missing VCS system information" error.  The script did not seem to do anything with this value though.  I just changed the form to allow me to edit the Name and entered a value of "TEST".  That got past the validation and it was able to discover repositories in my github account.
• The second issue is that it appears to require that the Terraform files are in the root directory of the repository.  I have a github repo that has several directories, with each project representing a Terraform project.  This is supported in Terraform Cloud by specifying a Working Directory on the Workspace.  For now I created a new repo with one Terraform project.  That worked, but I am not sure it is a long term solution.

Thanks again for your help (and @aniket.singh@servicenow too).