Data Stream Errors on Empty Rest Response

[Brian G]

Currently I'm creating a flow that has 2 API steps

1. First step there is an API call to pull a list of projects from google cloud

2. For Each project from step 1 a rest call needs to be sent to pull each Project record's Accounts

This step requires pagination to be used so a Datastream is being used for this.

I have this partially working, but am running into a rather puzzling issue.

if a project has accounts under it the response looks like:

{
"accounts": [
{
"name": "projects/prj-infrastructure-global-001/serviceAccounts/sa-grafana-sandbox@prj-infrastructure-global-001.iam.gserviceaccount.com",
"projectId": "prj-infrastructure-global-001",
"uniqueId": "102034640071361008578",
"email": "sa-grafana-sandbox@prj-infrastructure-global-001.iam.gserviceaccount.com",
"displayName": "sa-grafana-sandbox",
"etag": "MDEwMjE5MjA=",
"description": "reads metrics from cloud monitor",
"oauth2ClientId": "102034640071361008578"
},...etc

]

However, if a project doesn't have any accounts under it the response body will be empty: {}

I have my splitter step configured to look for $.accounts, but this fails with "Could not find path in stream: $.accounts" when i hit project with no accounts and the response is empty.

What options do I have to code around a empty response situation like the above?

[saichary]

Hello @Brian G 

Try building the endpoint so that during the initial REST API run, the query condition uses a WHERE clause with dot-walking to fetch only those projects that have associated accounts.

This approach will help reduce runtime, ensure you are working only with valid projects, and minimize the number of REST calls.

Please let me know if that works

[Brian G]

While this could be a solid solution where available i don't believe the google API provides account information at the get project api level. I did not see where this was possible on https://cloud.google.com/resource-manager/reference/rest/v1/projects/list. This is why we first need to get the list of projects to know what they all are, then do a second api call for each to get more details on if they have accounts.

[saichary]

@Brian G

You can adjust your approach like this:

1. First, get all billing accounts using https://cloudbilling.googleapis.com/v1/billingAccounts.

2. For each account, fetch its projects using https://cloudbilling.googleapis.com/v1/billingAccounts/{ACCOUNT_ID}/projects. This way you get all projects linked to that account in one call.

3. If you also need account details like display name or status, use https://cloudbilling.googleapis.com/v1/billingAccounts/{ACCOUNT_ID} (only once per account).
   
   Please let me know if that works, if this resolves your issues please mark this answer as helpful
   
   
   

[palanikumar]

Can you explain how you implemented the splitter step