Email sending limit

[Katie A]

We had an issue recently where a Business Rule got caught in an infinite loop and triggered several thousand events.

Now, my question is there a way to set an email limit?

There should never be a situation where the system is sending out more than a hundred or so emails at a time.

Much less the 60,000 emails that were sent out to users the other day.

I asked ServiceNow support and they told me that there is not an OOTB solution but might be achievable by writing a series of business rules.

Let me know if anyone has any suggestions.

Thanks!

[coryseering]

Hi Kathryn,

There are two properties you can use to control this:

glide.smtp.one.chunk.per.send (default is false)

glide.smtp.max.per.job (default is 1000)

The system works like this (as far as my recollection goes, I haven't looked at the SMTP code in a while):

A sys_trigger job runs periodically, and checks the sys_email table for outbound email. If outbound mail is found, it marks those emails as "claimed"- the system is going to try sending them out.

It then processes the outbound mail in batches (chunks). It will send out 100 emails at a time with brief pauses between chunks, (to close and recreate the SMTP connection) up to a maximum defined in the glide.smtp. max.per.job property. Once that max is reached, any emails that were claimed but are not sent get "unclaimed" and set back to send-ready.

The SMTP job then sleeps until the next time it's activated by the sy_trigger record.

You can reduce the total number of emails it will send per job by either forcing it to send only one chunk (in this case, 100 emails) and then shutting down for 1 minute (the default period between runs), or you can change the max per job setting to a lower number (in this case, 1000). In either case, you are reducing the flow of outbound emails from your instance, which can cause other problems.

Some instances have multiple SMTP jobs. I have two on my test instance, which means on average, every minute I could be sending out up to 2,000 emails. If you have multiple jobs, you need to take that into account when setting the values in those properties.

There are other things you could do to help with problems like this. For instance, a job that periodically checks the size of the sys_email table, and if there are a large number of send-ready messages, it could send a REST message to an outside notification service (you wouldn't want to drop an e-mail into the queue to tell you that the queue is big, obviously).

Or you could have a scheduled cron job on an outside system that polls xmlstats.do for your instance (it must provide admin credentials via basic auth) and check the iostats for the sys_email tables (or sys_email_log) for a large number of inserts. If you want to go that route, I suggest limiting the output of xmlstats.do to just the iostats:

/xmlstats.do?include=iostats

I am sure that some other customers have clever methods for email monitoring. Hopefully some of them respond with how they look for large, unexpected increases in email volume, and how they determine whether it's legitimate traffic or not.

I hope that's helpful,

Cory

[Katie A]

As always you're extremely helpful with your thorough responses. Thanks again Cory!