Data Stream Errors on Empty Rest Response

[Brian G]

Currently I'm creating a flow that has 2 API steps with the goal of pulling all Service Accounts  for our population of projects:

1. First step: Identify Project population aka get all projects. An API call to pull a list of all projects from google cloud

2. Since Service Accounts are housed under Projects in a 0-many fashion, Another API call is needed for Each project pulled in step 1. This rest call pulls each Project record's Service Accounts data.

This step requires pagination to be used so a Datastream is being used for this.

I have this partially working, but am running into a rather puzzling issue.

if a project has accounts under it the ServiceAccounts API response looks like:

{
"accounts": [
{
"name": "projects/prj-infrastructure-global-001/serviceAccounts/sa-grafana-sandbox@prj-infrastructure-global-001.iam.gserviceaccount.com",
"projectId": "prj-infrastructure-global-001",
"uniqueId": "102034640071361008578",
"email": "sa-grafana-sandbox@prj-infrastructure-global-001.iam.gserviceaccount.com",
"displayName": "sa-grafana-sandbox",
"etag": "MDEwMjE5MjA=",
"description": "reads metrics from cloud monitor",
"oauth2ClientId": "102034640071361008578"
},...etc

]

However, if a project doesn't have any accounts under it the API response body will be empty: {}

I have my splitter step configured to look for $.accounts, but this fails with "Could not find path in stream: $.accounts" when i hit project with no accounts and the response is empty.

There does not seem to be a way to gracefully ignore these failures and move on and no scripting is available between the REST step and the splitter step in these data streams which seems like an oversight for this type of issue.

What options do I have to code around a empty response situation like the above?

[saichary]

Hello @Brian G 

Try building the endpoint so that during the initial REST API run, the query condition uses a WHERE clause with dot-walking to fetch only those projects that have associated accounts.

This approach will help reduce runtime, ensure you are working only with valid projects, and minimize the number of REST calls.

Please let me know if that works

[Brian G]

While this could be a solid solution where available i don't believe the google API provides account information at the get project api level. I did not see where this was possible on https://cloud.google.com/resource-manager/reference/rest/v1/projects/list. This is why we first need to get the list of projects to know what they all are, then do a second api call for each to get more details on if they have accounts.

[saichary]

@Brian G

You can adjust your approach like this:

1. First, get all billing accounts using https://cloudbilling.googleapis.com/v1/billingAccounts.

2. For each account, fetch its projects using https://cloudbilling.googleapis.com/v1/billingAccounts/{ACCOUNT_ID}/projects. This way you get all projects linked to that account in one call.

3. If you also need account details like display name or status, use https://cloudbilling.googleapis.com/v1/billingAccounts/{ACCOUNT_ID} (only once per account).
   
   Please let me know if that works, if this resolves your issues please mark this answer as helpful
   
   
   

[palanikumar]

Can you explain how you implemented the splitter step