Transform map onStart script to check if trailer exists in csv file, if not abort the transaction.

Lavanya6
Kilo Contributor

I am running Transform map to insert and update user records. Data source is a csv zipped file. My logic is not working when i am checking for trailer in the source.

Requirement is:

1.If the source field does not have trailer information (as in the attachment) the transaction should get aborted that is the transform map should not happen.

2.If the record count from source does not match the record count in the transform map table the transaction should be aborted.

1 ACCEPTED SOLUTION

Hi Lavanya,

here is the script; I tried in onStart transform script but somehow it didn't work out.

It worked well when added in onBefore transform script

script below

Note: Ensure you give proper

1) import set table name; I have used u_testing_data_load_csv

2) correct import set field name; -> u_recordcount and u_h

(function runTransformScript(source, map, log, target /*undefined onStart*/ ) {

	// Add your code here
	var trailerCharacter = '';

	var recordCountField;
	var gr1 = new GlideRecord('u_testing_data_load_csv');
	gr1.orderByDesc('sys_import_row');
	gr1.addQuery('sys_import_set', source.sys_import_set);
	gr1.query();
	if(gr1.next()){
		recordCountField = gr1.u_recordcount;
		trailerCharacter = gr1.u_h;
	}

	if(trailerCharacter != 'T'){
		log.error('Missing trailer row as the character T cannot be found in column H');
		error = true;
	}
	else{
		// found the tralier row now check whether the count of rows is same as record count column

		var gr = new GlideRecord('sys_import_set_row');
		gr.addQuery('sys_import_set', source.sys_import_set);
		gr.query();
		var totalRowsIncludingTrailer = gr.getRowCount();

		var actualDataRows = totalRowsIncludingTrailer - 1;

		if(actualDataRows != recordCountField){
			log.info('Total rows are not matching hence setting error as true and it would halt entire transformation');
			ignore = true;
			error = true;
		}
		else{
			// allow transform to run
		}
	}

})(source, map, log, target);

Mark Correct if this solves your issue and also mark 👍 Helpful if you find my response worthy based on the impact.
Thanks
Ankur

Regards,
Ankur
Certified Technical Architect  ||  9x ServiceNow MVP  ||  ServiceNow Community Leader

View solution in original post

16 REPLIES 16

Thanks,

For the following line, what is the output?

log.info("import set ID is " + source.sys_import_set);

Hi Kieran,

I had to made a small change which returns me the import set number.

log.info("import set ID is " + import_set.number); 

 

Thanks

Hi Lavanya,

please comment by giving your response and this can be checked

1.If the source field does not have trailer information (as in the attachment) the transaction should get aborted that is the transform map should not happen.

- how you are identifying the trailer information? Is it that the last row in the csv should be having value as T in the 1st row. Also will the last row not have any information? Will it be just used for indicating it is the last row

2.If the record count from source does not match the record count in the transform map table the transaction should be aborted.

- the record count is the last column in the csv and the last row will have the final record count. Are you saying if they sent csv with record count as 25 then there should be 25 rows in the csv?

Regards
Ankur

Regards,
Ankur
Certified Technical Architect  ||  9x ServiceNow MVP  ||  ServiceNow Community Leader

Hi Ankur,

Thankyou for your response.

1. Yes the the last row will have trailer information. In csv file last row first column will have the 'T'. There will be no data/information in the last row. the other thing in last row last column will be the total number of rows or records. 'T' indicates that this is last row.

find_real_file.png

 

2. Yes, if they sent csv with record count 25, we need to check in servicenow before the transform happens if the record count matches.

 

Thanks,
Lavanya 

Hi Lavanya,

In the screenshot you shared is the below correct assumption?

a) they have sent the last row with starting column value as T so it should pass

b) the last row contains Record count as 4 so it means including the Trailer Row the count would be 5 i.e. Record Count + 1

Regards
Ankur

Regards,
Ankur
Certified Technical Architect  ||  9x ServiceNow MVP  ||  ServiceNow Community Leader