Performance issues across entire system all users

[Katie A]

We are having some very impactful performance issues in our instance. We are concerned because we are on the verge of populating our CMDB with multiple data sources -- but we aren't comfortable proceeding considering the system performance is already terrible. Most of the time pages are just very slow to load, but other times the page does not load at all and the browser crashes.

The issue affects ALL users in ALL areas of the system. The issue is affecting production as well as dev and test instances.

We are seeing client transaction times around 15 seconds, some as high as 20 seconds.

I opened a ticket with support but they gave us the run-around with insufficient answers, such as checking homepage refresh times. They were dismissive and tried to mark the incident as resolved without giving us any real solutions.

They checked out the health of our nodes and explained that everything seemed normal. We moved data centers about a month ago and we have requested moving back to the old data center to see if that might have been the cause of the issue.

Still, we are skeptical and extremely unhappy.

The issue cannot be homepage refresh times since the same issue is impacting both Dev and Test where there is only ONE user working there on a daily basis.We are a small company and we only have about 40 users in the system.There is no way that homepage refresh times are causing such response issues.

I checked all of the performance graphs on CPU, JVM memory, SQL transactions, etc. I don't see anything obvious in those graphs to indicate a problem.

Attached is a screenshot of the client response times which is very slow.

We are on Fuji Patch 10. The issue started about a month ago and has gotten worse in the past few weeks.

Has anyone experienced similar issues? We are not sure how to proceed considering the lack of real help from support.

[HugoFirst]

OK, in that case, it's time to make like Sherlock Holmes and sleuth around.

Turn on debugging for business rules ( detailed )

It will show how much time is spent in which business rules.

You can do likewise for queries by turning on debug SQL (Detailed)

Look for items which have unusual service times.

[Katie A]

Thank you Steve. I had been looking over the Performance meters on the homepage, but I will give that a try as well. Thanks again for the recommendations.

[JC Moller]

Hi,

If you click on the browser text, you can view more information about how time is used in processing the request in the browser.

Where are the 5 - 8 seconds used in your example?

- Jan

[HugoFirst]

We have a highly customized change application and we worked thru some performance issues a while back.

To paraphrase Churchill,   it will take "Toil and Sweat", but fortunately no "Blood or Tears"!

One thing that I noticed with our customization is that the same record was often updated multiple times in one "transaction".

Several Business rules have "current.update() in them and the change request ( or change task or whatever ) would be updated multiple times, when it really only needed to be updated once.

I was also surprised to learn that each time the record updated, the "update business rules" would run again!   There's a definite opportunity for infinite recursion here.   I wonder if ServiceNow has a limit to how deep you can go in an update transaction.

[gbleir]

Steve,

We have a similar issue.   When you say that "current.update()" could be a problem could you expand on that?   Is it because it is found in more than one business rule, or multiple time in a business rule?   For instance, we have a Business Rule "Set assignment group if null" that runs "after".   It contains the following:

current.assignment_group.setDisplayValue('Service Desk');

current.update();

Is there any reason that it would run more than once during a transaction?   We also have four or more Business Rules on the Incident table both "before" and "after" that update counts etc. using "rec.update();"   at the end.   I am trying to figure out what can cause the "multiple" updates that you mention.     Hope I wasn't too vague here!  

Thanks,

Greg Bleir

Broome Tioga BOCES