IMPORTANT! New SPAM configurations & SN to SN mail integrations

[Uncle Rob]

A customer recently notified me that they were missing a lot of important updates to tickets.   The updates were expected to be delivered via an email integration.

Checking sys_email, I noticed a ton of new received-ignored messages with failure messages like this...

The error string didn't provide any help, so I dug into the headers.   BAM!   All the failures had X-ServiceNow-Spam flags in the headers.  

That's critical info because of this default sys_property:

What troubled me is why this suddenly became a problem.   I managed to find KB0549426 which stated...
"Note that as of 6/29/2017, SPF checks now factor more heavily into a mail's spam score.   In order to be flagged as spam, a message must have an aggregate score of 6.2 or higher.   A soft SPF failure will add 6.0 to the score, whereas a hard SPF failure will add 7.0 to the score (immediately flagging it as spam).   It is recommended to check and ensure that your company's SPF records are correct and up-to-date, or some messages may be inadvertently marked as spam."

So that explains why inbound email actions have been so unreliable for the past couple weeks.   I recommend everyone check their sys_email logs for anything with "received-ignored" state.   You could be in my customer's position with a *TON* of missing data.

POSSIBLE RESOLULTIONS

- So the KB article instructs you to "update the SPF records", but this is coming from another ServiceNow instance.   Does this mean all SN to SN mail integrations are now suspect?

- Remove X-ServiceNow-Spam-Flag:YES from the "Ignore mail with these headers" sys_property.

[Uncle Rob]

As I learn more about what an SPF record is... the question I posed above becomes far more important:

Since this is a ServiceNow to ServiceNow mail integration, and I'm getting SPF soft failures.... does that mean that ServiceNow needs to update their own SPF records?

Asking for a friend.

[Uncle Rob]

WHAT TO LOOK FOR

Since the error string doesn't mention anything about the SPAM factor, I suggest you create a bookmarked filter on the sys_email table with the following conditions:
- Header CONTAINS X-ServiceNow-Spam-Flag:YES

- Type = Received-Ignored

[mike_zelnik]

Hi Robert,

I suggest you or the customer open an incident in HI so we can look at some examples and your specific configuration.   The support team is well aware of this recent change and some issues this has caused for customers.   I recommend mentioning in the incident these are SN instance to instance emails which changes the tenor of the issue.  

-Mike

[mike_zelnik]

Here is a known error article about some of the issues which could arise from the change and suggested workarounds:

https://hi.service-now.com/kb_view.do?sysparm_article=KB0623808

Note: You must login to HI to view the article.

[Tom Charly]

UPDATE :

As of 15 July 2017, following an internal change, the spam scores for failed SPF checks have again been changed by ServiceNow. From now on a soft SPF failure will add 3.0 to the score, whereas a hard SPF failure will add 3.5 to the score. Any score of 6.2 or higher will cause the email to have the email header X-ServiceNow-Spam-Flag:YES set, marking it as spam.

This change makes it *less* likely that emails, that fail SPF checks, will be marked as spam. Although SPF checks are a useful way to detect spam, we found that legitimate email, such as email relayed from other domains, could fail SPF checks as well which lead to modifying the rules again.

Thanks,

Tom Charly  

Network Integration Team - ServiceNow

[Uncle Rob]

UPDATE:  
How to be a ServiceNow Dev - Lesson 36:   When you've fully read a log, take a break, breathe, and read it again.

When I originally created this thread it appeared to me that ServiceNow's own mail sending capability was triggering its own spam identification on other instances.   After a careful second read through of the logs, it appears that the sending instance was going through one of their own SMTP servers.   As always, ServiceNow support was a massive help.   Fast & thorough!

It's still worth checking your mail logs for received-ignored mail, but the problem isn't ServiceNow itself.  

[mukulgupta]

Hi rfedoruk,

I am glad we were able to help!

Thanks!

[SusanWinKY]

Robert, we experienced this same thing this past February (2017) but only in our Production instance (sub-production instances were not impacted).   We were on Helsinki at the time.   Obviously, we did not make this change ourselves, so I must assume that SN pushed this update automatically.

This concerns me for a couple of reasons...

1.   We experienced it in February, several months before the date mentioned in the KB article.

2.   We didn't receive any advanced notification (i.e., HI ticket) about it as we do for other auto updates to our instances.

Is it typical that SN pushes updates to instances (separate from scheduled maintenance, patch, hotfix, or update)?

[Uncle Rob]

It is most definitely *NOT* typical of SN.   In the 11 years I've been at this, I struggle to remember a time when a rogue update from SN ruined my day.