Intermittent service request creation failures during bulk inbound email processing using Cart API

[bchandra]

: We are experiencing an intermittent issue where inbound emails are received and processed successfully, but some Service Requests are not created during bulk email processing.
Observed behavior:
All inbound emails are received successfully in the sys_email table.
The inbound email action is triggered.
During bulk processing, some emails do not create the expected Service Request (REQ/RITM), while others are processed successfully.
Reprocessing the affected inbound emails manually creates the missing Service Requests successfully.
The issue is intermittent and is primarily observed when a large number of emails are received within a very short time interval.
The current Inbound email action has been working successfully in production for 3 years without any issue.
During our investigation, we have identified that the implementation uses legacy Cart() API and have updated its to sn_sc_CartJS for validation. 

HAs anyone experienced this type of issue before?

[Vikram Reddy]

Hi @bchandra,

I'm going to assume your inbound email action is still calling Cart or CartJS synchronously, inline, inside the same transaction ServiceNow uses to process each sys_email record, and that a burst of emails means several of those transactions are landing within the same second or two rather than strictly one after another.

If that's the setup, switching from Cart() to sn_sc_CartJS won't fix this, because both APIs are built the same way underneath: they resolve to a persistent cart record keyed to a user, meant for one interactive session placing one order at a time. Neither was designed to be hammered concurrently by multiple background transactions. When email volume spikes, the platform processes that backlog with more than one worker in parallel to keep up, and if two of those workers resolve the same requested for user (or the same processing/integration user) at nearly the same moment, you get a race on that user's cart. One thread's addItem or checkout can get clobbered by the other without throwing anything, since the API doesn't hard fail loudly, it just quietly doesn't produce a request. That lines up with what you're seeing: no errors in the log, the email itself is fine, and reprocessing one at a time later works because there's no longer any contention.

A few things worth checking before you chase this further:

• Look at the requested for user resolved on the failed emails during the burst window. If it's a shared inbox, distribution list, or a common integration user rather than distinct individual senders, that's your collision point.
• Check whether your inbound action is defined to run as an Async Business Rule or is being queued through sys_trigger. Async jobs get picked up by whichever scheduler worker is free and don't guarantee execution order or isolation between records touching the same user.
• Wrap the addItem/checkout block with a lock keyed on the requested for user, for example a row insert into a small custom table with a unique index on that user, so a second transaction for the same user waits instead of racing. GlideMutex works too if you're comfortable with it.
• Add explicit logging right after checkoutCart or orderNow that checks the returned request/RITM value and writes to the system log if it comes back empty. Right now you likely have no visibility into the failure at the moment it happens, only the absence of a REQ afterward.
• While you're in there, check for orphaned sc_cart or sc_cart_item rows tied to the affected user around the failure timestamps. Leftover cart rows with no resulting request are a strong signal this is a concurrency collision rather than a data or mapping issue.

If the requested for user turns out to be different for every failed email, this theory doesn't hold, and I'd look instead at whether your instance's node count or scheduler thread pool changed recently, since "worked fine for three years, now doesn't" often tracks with an infrastructure or capacity change more than a script change.

Thank you,
Vikram Karety
Octigo Solutions INC