Enhanced scheduled jobs for Warehouse

[Sivagamasundari]

Create enhanced scheduled jobs for Warehouse for multiple jobs to be executed at the same time. Currently, only one job can be executed at a time due to a singular property. 

The jobs should be able to generate large amount of data and should automatically chunk them with correct names. Currently, it requires Admins to generate a file with only 500K rows, resulting in manual work for admins.

[Community Alums]

Hi @Sivagamasundari ,

You should be able to achieve this by writing an script for data generation and chunking the data.

First thing you need to remove the singular property and enable concurrent execution.

Also try to run the scheduled job at regular intervals automatically rather than at a particular time.

Pseudo code would be something like this-

SET CHUNK_SIZE TO 500000

FUNCTION processDataInChunks
CREATE gr TO getRecordsFrom('your_table')
SET chunkCount TO 0

WHILE thereAreMoreRecords(gr)
IF chunkCount IS GREATER THAN OR EQUAL TO CHUNK_SIZE
CALL handleChunk(chunkCount)
SET chunkCount TO 0
END IF
CALL processRecord(gr)
INCREMENT chunkCount
END WHILE

IF chunkCount IS GREATER THAN 0
CALL handleChunk(chunkCount)

FUNCTION processRecord(record)
// Do something with each record

FUNCTION handleChunk(count)
// Save chunk of data with a unique name

CALL processDataInChunks()

If my response has resolved your query, please consider giving it a thumbs up ‌‌ and marking it as the correct answer‌‌!

Thanks & Regards,

Sanjay Kumar