Data Source report error when data stream generate a blank payload.

[kevin33]

Good afternoon, all

I am wonder if anybody ever resolve this issue. I am really thanks for anyone able to provide directions. 

I have a integration set up like below:

REST API-> Data Stream Action-> data source->IH ETL. 

In data stream splitter steps, I am use item path. $.entities to extract pay load and pass to parser steps .

Normal pay load would like:

{
"totalCount": 11310,
"pageSize": 50,
"nextPageKey": "AQQTREAUS",
"entities": [
{
"entityId": "HOST-123456789",
"type": "HOST",
"displayName": "XYZ",
"firstSeenTms": 1665105217679,
"lastSeenTms": 1716399078945,
},
{
"entityId": "HOST-123456789",
"type": "HOST",
"displayName": "XYZ",
"firstSeenTms": 1665105217679,
"lastSeenTms": 1716399078945,
}
]
}

but some tine it return:

{"totalCount":0,"pageSize":50,"entities":[]}

When payload  like tis, the data source load data process report error, 

No fields provided for mapping to headers

I did try the following in splitter steps:

-Change item path from $.entities to $, but the payload seem not pass

-Try to use $.*, but got error message says wildcard is invalid

Seem all not working 

I am wondering how data stream handle the blank pay load...How can I pass the whole payload to parser steps.

Thanks Again. 

Thanks again

[Tony Chatfield1]

Hi, unfortunately your post does not make your issue or configuration clear.
Perhaps you could update this thread to include your script and example payload (both as plain text), so that the community is in a better position to review and advise.

[Andrew Grant1]

I seem to be facing a similar issue that Kevin33 was. I'll describe the issue below in an attempt to clarify:

I'm querying an external API and successful responses contain a .data property that includes an array of objects. Unsuccessful/erroneous responses lack this property and instead include a .error property with related information to the error.

Considering this and in an attempt to handle error/successful responses I had tried to use $[error,data] as item path. It is supposed to merge both .error and .data paths. So when considering that successful responses contain .data (and not .error) and erroneous ones contain .error (and not .data) I was assuming that this would return the information needed and that I could subsequently handle either the errors (or the data) within the script step following the parsing step.

Unfortunately it appears that SN is using a limited set of JSONpath capabilities relating to the declaration of an item path. While the item path described above worked on https://jsonpath.com/ when determining an effective item path based on my responses, SN did not appear to like it and stated that it was wrong.  Consequentially, an effective way to handle possible errors is elusive when using Datastreams. Hoping a future update is introduced that makes handling errors more intuitive.  Getting the whole payload and manually setting the output object would be ideal considering the issues faced.

[kevin33]

Hi, Andrew, Thank you for reply, 

I was running into anther similar issue recently. I have an external API call with paging for JSON payload, some call return just [], I notice when data stream encounter this issue, it stop for next page, and report completed. 

I am wondering if any one have any idea how to skip this kind payload and continue for next page. 

Thank you 

[Mike W_]

Hi @kevin33 , did you ever find a resolution to your issue? I'm having the same problem now. I have a datastream I'm calling in a loop within a flow, and as soon as one of the iterations of the loop returns 0 data (which I'd consider a valid thing to return for some iterations of the loop), the datastream action considers it an error, and the whole loop & flow stop processing. I want it to continue processing the remaining iterations of the loop.