Integration job doesn't always process input

[Karla Chorny]

I'm having trouble understanding why my integration job doesn't always complete when run on a schedule. I can see in the ECC queue that the request was sent and the data was received, but the data doesn't get processed. In the code below, is there an obvious reason this is happening?

If I run the job manually early in the morning, it works every time.

	function GetHRProfileData() {
	    try {
		gs.info('HR Profile: Import started');
	        var r = new sn_ws.RESTMessageV2('MyRestMessage', 'GET');
		r.setEccParameter("skip_sensor","true");
	        var response = r.execute(); //it gets this far
	        var responseBody = response.getBody();
	        var httpStatus = response.getStatusCode();
		gs.info("HR Profile: HTTP Status: " + httpStatus); //it doesn't get this far
	        this.processResponse(responseBody);
	    } catch (ex) {
	        var message = ex.message;
	    }
	}

function processResponse(r) { etc etc etc

[Karla Chorny · Accepted solution]

I went around and around with support on this and never got a resolution. They continued to blame transform errors, but when this issue occurred as described above, the transform never happened, therefore the transform errors are irrelevant.

After discussing with a colleague, I decided to move this from a scheduled job to a scheduled flow. I created a new action using the OOB REST function, and set the timeout to 45 seconds, then added the action to the flow. The flow is scheduled to run once per day, but we needed it to run twice a day, so I also created a scheduled job that will trigger the flow the 2nd time each day.

These updates are still in our dev environment being tested, but so far the data pull and transform is happening each time as expected. The ECC Queue messages changed (versus when they are created via the REST call in a scheduled job) to IPaaSActionProbe type, and the 'Source' field is the sys_id of the flow context record. When I view the output and input messages, they are frequently more than 29 seconds apart (the prior breaking point), indicating to me that the issue is resolved as long as the results are returned within the defined timeout of 45 seconds.