Transform script suggestion
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11 hours ago
Hello all,
I'm seeking your input on a scenario we're currently handling.
Current Flow
A third-party Talend application sends 5,000 records daily to a ServiceNow (SNOW) transform map.
In the onBefore script, we perform validation:
If any field is empty, we create an Incident (INC).
Sample Data
| 2345 | LinkDin | 2345LinkDin#abc | Open | 987098 |
| 8987 | Noukri | 8987Noukri#efg | Resolved | 453672 |
| 1212 | Freshers | 1212Freshers#asdf | Open | 543878 |
As shown, the Source field can be one of: LinkDin, Noukri, or Freshers.
New Requirement
Talend will continue sending records in the same format, but:
All records will have Source = Noukri.
Most fields will contain inaccurate data, except for the Private Key.
We need to:
Extract the Private Key.
Send it to another third-party application.
If valid data is returned, update the record with correct Source and other fields in import set table.
- If data is not returned keep the same data in import set.
Perform validation as usual.
Proposed Solution
Use a Script Action triggered by the onStart script:
Fetch all 5,000 records from import set.
Send them to the third-party application.
Based on the response, update the Import Set records.
Let onBefore/onAfter scripts handle validation and transformation.
Challenge
Processing all 5,000 records in the onStart script is time-consuming.
This may cause delays or stuck records in the Import Set, potentially cancelling the job.
Request
Could anyone suggest a more efficient approach to handle this scenario? Open to ideas around batching, asynchronous processing, or architectural changes.
Thanks in advance!
Would you like me to help draft a technical solution proposal or explore optimization strategies for this workflow?
