Slowness detected in On-Prem instance

[SagarJ957172966]

Dear Team,

We are on ServiceNow On-Prem instance with 3 App servers and total 15 nodes across the servers. 

We have seen slowness while accessing ServiceNow application. 

Please refer to below details;

OS Version :

NAME="Red Hat Enterprise Linux"
VERSION="8.10 (Ootpa)"

Database : Oracle 19c in cluster environment

I would appreciate if anyone can help me to resolve this issue.

[jeffreyluto]

Case into Servicenow usually assists into what setting need to be in the glide.properties. and glide.db.properties files in the dir: /glide/nodes/svc_16000/conf 

the DB files has credentials that are specific to our instance cannot share that:

The glide.properties files is very generic see output below

$ cat glide.properties

# glide.properties (for DB related properties, use glide.db.properties)

glide.proxy.host = http://hostname
glide.proxy.path = /
glide.servlet.port = 16000
glide.cluster.node_name = app1
glide.db.pooler.connections = 32
glide.db.pooler.connections.max = 32

glide.installation.self_hosted = true
glide.usageanalytics.central_instance = 'https://disabled.service-now.com'
glide.ua.downloader.central_instance = ''

override file I think is most important

$ cat 51-memory.properties
##
###

glide.java.opts.snippet.nodeconfig.mem=-Xms2048m -Xmx2048m -XX:MaxMetaspaceSize=640m -XX:ReservedCodeCacheSize=240m -XX:MaxDirectMemorySize=256m -XX:-UseAdaptiveSizePolicy

--- I would also try restarting the Database servers or renice some process there.

Some have helped us

[jeffreyluto]

Network latency ( this is from the self-hosted FAQs  ServiceNow)

ServiceNow's standard for network latency between App -> DB servers is a minimum ICMP ping time of less or equal to.250ms. Many self-hosted environments will deploy network gear inline between the App -> DB servers which at time incur network latency. Some of these devices might be firewalls, IPS/IDS appliances or even proxy servers. Some customers have found that installing the App servers in the same network (on the same LAN) has solved any latency issues.

It's recommended your network latency is monitored to ensure the latency stays at or below .250ms. It's also highly recommended to have in place monitoring for packet loss. We've seen sever performance impacts to environments where switch gear starts failing and packet loss is one of the side effects.   

Here are some things I would suggest and at times worked for us.

-Consider putting DB and App in the same subnet

-Consider creating specific routes or QoS for this traffic

-Consider creating a different route via another vNIC than the inbound from the load balancer?

-Double-check Iptables/Firewalld

-beware of Network proxies

-Run traceroute and ping to see where and how long traffic take and adjust