lRobust Transformer Scheduled Data Import Transaction cancelled: Available memory is almost depleted

[MarcinM]

Hello Developers!

I'm struggling with a big data load (~170k records) that always ends with: "Transaction cancelled: Available memory is almost depleted".

Configuration:

ETL Definition should populate 4 tables

Robust Import Set Transformer Batch size is set to 3000

Scheduled data import is partitioned to 15000 chunks

Output while transforming:

First two partitions (15k each) were transformed correctly but the other ones had completed with errors. I checked in the ISETs and the errors were the same: "Transaction cancelled: Available memory is almost depleted".

What can I check to define what is trully causing this error? Initially I was thinking that maybe the source file is somehow corrupted so I grabbed the last processed entry and couple of next ones and created a source file by hand counting that I could reproduce this error but with no luck.

I'm counting on your suggestions!

Take care,

Marcin

[DrewW · Accepted solution]

Have you tried to make the batch size smaller and/or import smaller chunks?

[MarcinM]

Hello @DrewW ,

thanks a lot. Indeed setting Batch size to 2000 did the trick.

Regards,

Marcin