Unable to load ServiceNow Instances (Prod,QA,Dev)

[SaiVasanth]

Hello Everyone,

All of a sudden I'm unable to load all the 3 ServiceNow Instances. Is there anyone who is facing the same issue?

Thanks & regards,

Vasanth.

[rajesh44853]

Connection is Sporadic. Its bit slow but looks to be working

[Fash]

It's down in many regions in Canada as well. Impacted ISPs are Bell and Rogers (So far).

[Gino Legacy]

This is the last update we received from ServiceNow stating service has been restored to the affected instance.

Case Action Summary

Est. Impact Start Time: 7:33 PM UTC, 2024-08-26

Est. Impact End Time: 9:43 PM UTC, 2024-08-26

Relief:

Service has been restored to the affected instance.

Summary:

ServiceNow has detected a potential DDoS event that may be impacting connectivity to your instance. Some users may be experiencing packet loss, performance degradation and intermittent availability when accessing your ServiceNow instance. 

Actions taken:

ServiceNow Subject matter experts have mitigated the DDoS event and currently monitoring. Initial indications are that this has reduced traffic and improved connectivity into your instance. We will update this case with additional details as information becomes available. We apologize for this inconvenience; your satisfaction is our top priority.

Most Probable Cause:

The most probable cause appears to be an issue relating to an DDoS event. Our investigation is ongoing. 

Next steps:

ServiceNow subject matter experts (SMEs) will continue to work on root cause analysis and provide further details once confirmed via the case. As relief has been provided, this case has been marked for Temporary Relief. This represents the final case action summary relating to this event. Please update this case should you have any further questions regarding this issue.

[Peter Sikari]

I logged a ticket with Support earlier today and I've just been notified of a "potential DDoS event that may be impacting connectivity to the datacenter where instance is hosted". 

Haven't been notified of an ETA as yet.

[Community Alums]

This is the latest update we received from ServiceNow.

ServiceNow will provide a root cause analysis (RCA) with more details of the event via this case no later than September 2, 2024.

Executive Summary

On August 26, at 19:22 (UTC), the ServiceNow network traffic management platform detected and began the automatic mitigation of a significant and sudden increase in network traffic that caused externally facing network circuits in the Phoenix, Arizona (PHX), and Toronto, Ontario (YTZ) data centers to become saturated due to a Distributed Denial of Service (DDoS) activity. The saturation of network circuits caused instances located in the affected sites to become externally inaccessible to most users. Additionally, some customers were unable to connect to their instance, NowSupport, or additional ServiceNow resources via the impacted data center regions, even if their instance was hosted in another data center location.

Network traffic that exceeded ServiceNow's automatic mitigation was manually redirected by ServiceNow engineers to a third-party network traffic scrubbing service provider to provide relief for most customers at 20:30. A small subset of customers remained impacted after the third-party network traffic redirection because of network latency and packet loss that was introduced during the traffic scrubbing process. An Advanced High Availability (AHA) transfer was executed for some of the remaining impacted instances to provide immediate relief while the remaining impacted customers were provided relief after network traffic was reverted to ServiceNow from the third-party service provider at the conclusion of the DDoS activity.

The event was caused by a large-scale DDoS activity directed at ServiceNow data centers.

ServiceNow has not observed any data loss or unauthorized access to instances resulting from this event. An AHA transfer of impacted instances was not automatically executed at the start of the event because throughout the event, all impacted instances remained online and were able to process data, and internal network routes remained healthy. ServiceNow engineers did not manually perform a mass AHA during the event because ServiceNow didn't want the DDoS activity to spread to other nonimpacted data centers.

To reduce the likelihood of customer impact during future DDoS events, ServiceNow configured network routes to automatically redirect network traffic that exceeds its designed throughput to the third-party network traffic scrubbing service provider. ServiceNow also performed network configuration changes to prevent network latency and packet loss from occurring during the network traffic scrubbing process.