Transform script suggestion

[sainath3]

Hello all,

I'm seeking your input on a scenario we're currently handling.

Current Flow

• A third-party Talend application sends 5,000 records daily to a ServiceNow (SNOW) transform map.

• In the onBefore script, we perform validation:

  • If any field is empty, we create an Incident (INC).

Sample Data

Product Number Source Reference ID State Private Key

2345	LinkDin	2345LinkDin#abc	Open	987098
8987	Noukri	8987Noukri#efg	Resolved	453672
1212	Freshers	1212Freshers#asdf	Open	543878

As shown, the Source field can be one of: LinkDin, Noukri, or Freshers.

 New Requirement

• Talend will continue sending records in the same format, but:

  • All records will have Source = Noukri.

  • Most fields will contain inaccurate data, except for the Private Key.

• We need to:

  1. Extract the Private Key.

  2. Send it to another third-party application.

  3. If valid data is returned, update the record with correct Source and other fields in import set table.

  4. If data is not returned keep the same data in import set.

  5. Perform validation as usual.

 Proposed Solution

• Use a Script Action triggered by the onStart script:

  • Fetch all 5,000 records from import set.

  • Send them to the third-party application.

  • Based on the response, update the Import Set records.

  • Let onBefore/onAfter scripts handle validation and transformation.

 Challenge

• Processing all 5,000 records in the onStart script is time-consuming.

• This may cause delays or stuck records in the Import Set, potentially cancelling the job.

 Request

Could anyone suggest a more efficient approach to handle this scenario? Open to ideas around batching, asynchronous processing, or architectural changes.

Thanks in advance!

Would you like me to help draft a technical solution proposal or explore optimization strategies for this workflow?

[Ricardo26]

Hi,

From what I undestand,  you don't need to send all the 5000 private keys in the imports. You actually just need to send the request to the third-party system when the private key is updated or a new record is inserted.

Now if the api supports getting the data of many private keys at once, you can create an array on a onStart script, and in an onAfter script make the comparassion between the current and previous private keys. If they are different, add in the array. Then in a onComplete, triggers the integration and update the fields.
Other option could be create an async business rules that will trigger the integration when the private keys changes.

[Ankur Bawiskar]

@sainath3 

Doing the validation on chunk of 5000 records in onStart transform script will lead to performance impact as you will call 3rd party API for those 5000 records and then perform the further processing.

I will suggest to inform the 3rd party team if they can send the accurate data and they can check the data in external API rather than having it done in ServiceNow platform.

💡 If my response helped, please mark it as correct ✔️ and close the thread 🔒 — this helps future readers find the solution faster! 🙏