Managing outbound email to Exchange Online mailboxes

[stevemac]

Our organisation is moving email to Exchange Online (EXO).  We have had a couple of issues where outbound email notifications from ServiceNow have exceeded the EXO inbound limits - both from single sender and to single recipient (EXO Send and receive limits).   The 2nd occurence caused a lot of NDRs to be sent the the ServiceNow instance mailbox, which tripped its limits / started responding with mailbox full (which caused issues for other legitimate email)

Both issues were caused by incident flood via integration (e.g. 3000+ incidents to same group in short period of time and the group had a lot of members so 100K+ emails sent).   Whilst the cause of the floods have been reviewed and better deduplication done before incidents are raised in ServiceNow, it is only a matter a time before we get another flood

The main culprit is the notification "Incident Assigned to Group".  

• Will probably ask users to disable the notification or set filters to reduce the amount of email notifications being sent.
• Where teams have group mailboxes, will encourage them to not also send notifications to group members

any recommendations on the following

• options in ServiceNow to throttle outbound email from ServiceNow (do not believe there are any)
• options to use a mail service between ServiceNow and the recipients that provides a solution to deal with EXO limits (not sure if this is exists or is possible)
• how to disable notifications to individuals / groups if outbound email is approaching EXO limits (not sure how to detect this before the outbound email is in the mail queue)
• anything else that you would think would assist to prevent recurrence

thanks,

Steve

[Kevin Moore]

We are running into the same issue. Hopefully someone can provide some insight to this issue.

[nataliya_b]

hi @Kevin Moore , @stevemac - how did your company resolve this issue? would be possible to share the solution?

[stevemac]

@nataliya_b, @Kevin Moore 

I'll say not solved fully, but currently in a better position

• created additional Exchange Online (EXO) mailboxes that are forwarding to the ServiceNow instance
  • one for approvals
  • total of three other ones for just about everything else
• updated notifications so load is better spread across the EXO mailboxes
• updated some notifications so that their from and reply to addresses are the instance email address
• updated mailto settings on instance so mailto address is set to the instance email and does not route back via EXO mailbox
• updated custom jobs / flows which we know to create a lot of email to do it in batches so to reduce likelihood of hitting EXO limits

The biggest contributor to our email is the Incident assigned to my group notification

• still working on temporarily stopping this notification to specific groups if we detect incident flood activity (and therefore a lot of email
  • plan is to look at incident metric table to see how many incidents have been assigned to each group in last X minutes and update the group record to block them (an remove the block in next check if unde the threshold)
  • the notification is updated to refer to setting on group record

Also looking at group records and challenging teams that are sending email to members and to the group email address (and the group email address is actually a distribution list)

Our EXO team is still not happy about the auto-forwarding, so I am expecting to make further changes.  This will most likely be using additional "From" addresses in notifications.  The "from" address will not be an actual mailbox.   Any replies to it will be dropped.  Reply to may be overridden to use the instance email.  Notifications will be updated to use the new addresses

• ServiceNowIncidentGroupNotification_P1@example.com
• ServiceNowIncidentGroupNotification_P2@example.com
• ServiceNowVulnerabilityGroupNotification@example.com

FYI

• DKIM configured for mail from ServiceNow that is using our domain
• internal teams engaged to ensure mail from new addresses processed through SPAM filters OK
• users communicated to
• reviewed email logs and bounced address records.  Found some users that had blocked the instance email address as it was not from our domain 

Good luck

[nataliya_b]

@stevemac - thank you very much for your detailed answer!