Designing for Performance Part 1

[Chris Pearson]

I'd like to bring some great information which I had the privilege of receiving from a recent ServiceNow webinar on the topic of designing for performance. Shout out to Haret Valean and Jon Gittoes from ServiceNow's Account Escalation Engineering (AEE) team. These are the people that get to the root cause when you can't figure out what in the world is making your SN instance drop to its knees and beg for mercy. Hint...it's usually you. But when we know better, we do better, right?

This might end up being a multi-blog series, but let's tackle the first topic.

The high level architecture can be described as an http request hitting SN's firewall, processing through an application delivery controller (ADC) which routes the request to multiple application nodes which then interact with the database layer. All SN instances have at least 2 application nodes. Each node has 2GB of Heap Space (memory), pools of threads which can execute work, and a JDBC driver with one or more pools of database connections.

Where does the AEE team see common bottlenecks in the SN instance architecture?

The first is with the primary database. If the database is slow it can effect the entire instance. This is because the primary database is typically a single shared resource and can be overwhelmed by excessive transactions (CPU / IO / History List Length). What can be done to mitigate these types of issues? For one, formulate your queries for optimization by avoiding as much as possible OR conditions and CONTAINS operators. Also, ensure that your tables are properly indexed. Queries against indexed columns will run extremely faster than non-indexed columns. When you start to troubleshoot your performance issue and you see spikes or abnormally high trends on SQL response times in your transactions, these are the things to look for.