Hi Everyone,
We are currently integrating a third-party vulnerability scanner with the Vulnerability Response (VR) application. We are using IntegrationHub (REST Action) to fetch the data.
The third-party API streams the results in NDJSON format (newline-delimited JSON), and the payload is massive.
Example Payload:
Content-Type: application/x-ndjson
{"platform_id":"abc123","asset_name":"prod-web-01","cve_id":"CVE-2024-12345"}
{"platform_id":"xyz789","asset_name":"db-server-02","cve_id":"CVE-2023-99999"}
The Problem: Currently, in our REST Step, we are using the 'Save As Attachment' option to save the payload to the "sn_vul_integration_process" table.
Because the incoming data is huge, processing this massive attachment downstream causes memory issues, and we cannot parse it efficiently.
Questions:
What is the best practice to handle large NDJSON streaming data in ServiceNow?
If we must use attachments, how can we chunk or split this streaming ndjson data effectively within the VR framework?
