Problems with sending SOAP messages asynchronously

[robbika]

I've come across a problem with sending asynchronous SOAP messages.

I've noticed this because I closed a Catalog Task that would close a Requested Item and a Request, that I knew would send out a SOAP message to another system (a B2B link with another system). The message is constructed and sent in a Business Rule. The relevant part of the script in the Business Rule is:

var response = s.post(true);

where 's' is the SOAP message and using s.post(true) means the message should be asynchronously inserted into the ECC Queue and then something else on the ECC Queue would deal with it. So, changing the status of a ticket thus causing a message to be posted to the ECC Queue, the web page should return immediately. In a synchronous interaction, I'd expect the web page not to return until the SOAP message has been sent synchronously.

What seems to happen currently, even though we are using 'post(true') is that the web page does not return until the SOAP transaction has happened or at least fails.

I know this because I put a ticket to closed, hit 'save', and the page did not return for more than 2 minutes. Curious about this, I looked at 'stats.do' on another page and looked at the currently executing Java stacktrace, and it was clear that the code was waiting for a SOAP response. Looking into the ECC Queue, indeed the message appeared there, and indeed failed with a timeout.

Questions:

- are my assumptions true? Should 'post(true)' put a message on the ECC Queue and then return leaving the queue to process it asyncrhonously?

- the Business Rules that all send message are 'after' BRs. Should they me 'async'? (If so I have some issues with that with journal fields maybe).

Has anyone else seen this behaviour? Are we doing something wrong?

Because of this behaviour we have many many issues. We recieve SOAP messages from systems that cause ticket to be created or ticket states to changes, sending out SOAP message to other system. This is all supposed to be event driven and asynchronous. If part of it isn't, runs synchronously, and has a timeout, then part of the overall system is going to fail, which indeed it is does on occasion.

This is in Dublin with the latest hotfixes.

[robbika]

This is not a simple problem.

If I make SOAPClient biz rule 'async' then I have given up the property that messages are processed in order.

It seems this is really a non-trivial problem to solve in SN.

[Dan117]

Async rules do run in order, but the underlying record could be modified between when the asyc rule was created and when it runs. Events could be fired from an after business rule so you have access to the previous object and to pass in specific information that could be changed such as journal fields, those events are then picked up by script actions.

It sounds like you are trying to do an integration of some type, and it comes down to what you need to guarantee in all of your calls. One thing that can be done is to create a separate message table and run the events off of that table. This also allows easy review and history for the integration.

Here is a good example of what I'm referring to: http://www.snc-blog.com/2013/02/04/sample-custom-incident-soap-interface/

[robbika]

Yes we have integrations in both directions.

We have B2B customers where tickets are coming in from our customers, and 3rd party resolvers get sent tickets from us, for fields services for example.

What is happening at the moment is that a field service engineer closes a ticket on their system, so SN processes an incoming SOAP messages, closes the relevant ticket, which causes a B2B ticket resolve message to be sent to our customer system. But the B2B messages has a timeout. Which means the initial incoming SOAP call from field services also has a timeout.

Then at this point all the 3 systems are out of sync. Field services had a timeout towards SN so it doesn't know what happened to the SN ticket. SN has closed the ticket. The message towards the customer had a timeout so likely the ticket on their side is not on closed.

Message timeouts causes a heck of a lot of problems. We currently spend a lot of time checking whether the tickets in the 3 systems are synced.

There are two problems:

- the system doesn't cope with message timeouts very well

- why are there timeouts in the first place? Each message on all systems should be easily processed in a few seconds

To stop the timeouts across 3 systems, I propose using full async message processing. So field services send a message, it is processed immediately and doesn't wait for SN to send the message to B2B. So field services will be in sync with SN from that point of view. Now if the message going out to B2B has a timeout, then at least it won't affect field services. But being 'async' with timeouts, messages could go out of order.

It is increasingly looking like I will have to implement some special message processing solution that fulfills all the constraints I have and to keep the systems in sync and recover from timeouts.

We are looking at the source of Java socket timeouts on message sending from SN as a different thread. No idea on that one at the moment.

[poyntzj]

Could look at it another way.

I have a BR that runs on each form and establishes if a SOAP action or action(s) need to be sent and calls a Script Include each time

For the one I am working on at the moment, I have to check to see if a number of fields have changed (excluding work notes / comments)

If needed, I then send that update over

Once that has come back as updated, I then send over any work notes / comments

Once that has come back as updated, I then have to send over another update to change a status (due to a limitation on the 3rd party side at the moment).

If at any point there is a problem the process stops and the user is told, otherwise they are informed of each update having completed.

For us it would be rare for the RITM to be assigned to the same 3rd party as the Request,but we have checked that and had no problems with the RITM was closed and updated the 3rd party while at the same time the Service Request BR fired and closed their Service Request as well.

[robbika]

What we did in the end is to make the SOAPClient business rule async with a current.update() on the end.

This decouples an incoming message from the outgoing message.

Business rules still put the message on the ECC queue syncrhonously so the data in the message is correct, but the sending is asynchronous.

This has worked well so far in that the timeout issues of a system sending us a message have gone away.

It is still possible for async message sending to be out of order. If there are two messages on the queue and one of them has a timeout, then it will go on to the next one.

Likely the next iteration of this will be to have a workflow that actually sends the messages in order from events in a queue. One workflow per tickets that keeps the message sending per ticket in order, but the workflows can run in parallel per ticket becuase we don't care about the order of messages between tickets.