Dedicated event processor

Ram117 · ‎05-06-2022

Experts,

Today I am working on an integration with a third party application which makes some POST call to create some CI records on the other application. Since the number of records I am looking at is 100k+ , I did not want to go with the workflow / f low based approach.

I am using the script solution wherein each records will create events and script actions will make the POST API call.

Now the question I have is, I did not want these events to block the default events within the platform, hence created a dedicated event processor to process these custom events. Now while setting up the sys_tigger record, I have a question on what needs to be selected as the 'system ID'.

If I do not set a value there, I see only 1 event processor getting created
If I set the value as 'Active Nodes', i see the multiple trigger records getting created ( for each node 1 )

can you guide me on this ? what is the ideal configuration which needs to be put in so as to not cause any havoc to default platform events ?

On a side note, I am expecting my events to grow in the future. May be b/w a 1M - 2M + to be created based on the number of CI's I am going to be processing..

thx

ram.

Allen Andreas · ‎05-06-2022

Hello,

From my understanding and when I've used this, multiple records are created on your active nodes. What I believe the support documentation is saying is that the delegated process out of box for normal event processing, takes up to 500 events and then stops.

For your custom event, it's going to grab the entire queue...no matter how many is there, at that time.

However, your jobs on the other nodes will also do the same, and they don't run at the same exact second as each other.

So when node 1 scoops up the entire queue, once it moves on, more events are still pouring in, then node 2 goes and scoops it up and moves on, etc.

I didn't say it round robins it, I said it spreads them out. That's what I meant by my above reply.

It would be pointless to select "Active Nodes" then if the way you're thinking it works, actually worked, haha. Because if it creates 3 event processor jobs on 3 nodes, but only 1 runs...which one is it? The parent record doesn't actually run, the child jobs run, and the child jobs are on the active nodes.

Good conversation either way, feel free to submit a case to SN support and double-check with them as well, if needed 🙂

Please mark reply as Helpful/Correct, if applicable. Thanks!

Please consider marking my reply as Helpful and/or Accept Solution, if applicable. Thanks!

View solution in original post

Allen Andreas · ‎05-06-2022

Hi,

The guidance as presented in this support article is what you should follow: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0676909 - for the scenario you've mentioned. If you can work to filter the data coming in not be daily, but a weekly refresh, etc. it's recommended to consider that approach instead of live feed (for everything).

Please mark reply as Helpful/Correct, if applicable. Thanks!

Please consider marking my reply as Helpful and/or Accept Solution, if applicable. Thanks!

Ram117 · ‎05-06-2022

Thank you for the response Allen.

I had looked at the article prior to posting. It doesn't explain the significance of have the system ID set to 'Active Nodes'. Does it mean that, each active node will have a custom event processor created , which will parallelly process these custom events based on when it gets the resource ? Can you please explain that bit ..

Allen Andreas · ‎05-06-2022

Hi,

If you can, please state things like that: what you've referenced, etc. else we don't know?

With that said, it means it's going to run that job on active nodes. So it'll create multiple records, one for each node you have as active.

If you only did it for one node, then all of your events would be processed on just that one node and it creates a bottleneck. If you spread it out amongst your other nodes, then it lessens the burden on a single node.

Please mark reply as Helpful/Correct, if applicable. Thanks!

Please consider marking my reply as Helpful and/or Accept Solution, if applicable. Thanks!

Ram117 · ‎05-06-2022

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0676909 ,

this article states that, custom event processor do not distribute the load. So doesnt it mean that, even if I set it to 'Active Nodes' ( 1 processor for each node ) one single event processor would claim the entire events in the READY state ? .

It makes me think, if I need multiple event queue to support my requirement to distribute the load ..