Real Time Data migration from Database to CMDB

SujanD345956992 · a week ago

An environment involves Point of Sale devices where agents are installed on each target devices to get active status of devices through Agent Client Collector. All the transactions that are made in POS / POC devices will get stored in separate Database.

How can we bring the transaction data from separate Database of the target POC device to Service Now CMDB securely without MID Server Path?

pavani_paluri · a week ago

Hi @SujanD345956992 ,

To securely bring transaction data into ServiceNow without a MID Server path:
- Expose the POS transaction database through a secure API layer.
- Use REST APIs + Import Sets or an integration platform to push data.
- Store transactions in custom tables linked to CMDB CIs, rather than directly in CMDB.
- Secure the integration with HTTPS, OAuth, and role‑based access.

Mark it helpful if this helps you to understand. Accept solution if this give you the answer you're looking for

Kind Regards,

Pavani P

Mark Manders · a week ago

https://noderegister.service-now.com/kb?id=kb_article_view&sysparm_article=KB0815247

High-Level Architecture without MID

Agent Registration

When using ACC with DEX or ACC-V (MID-less), mTLS authentication is used to establish a secure connection between ACC and the ServiceNow Cloud. This is a multiple step process that involves several steps. If any step fails due to network or instance configuration, the agent will not successfully register. See "Special network considerations" below for more information.

The Agent registration process involves the following steps.

On the customer instance, a registration key is automatically generated for the Agent Client Collector (Agent).
The Agent is installed in the customer instance using the registration key, instance URL, and public endpoint.
The instance URL corresponds to the INSTANCE_URL variable in the one-line installer command. The public endpoint refers to the DNS name of the nearest ServiceNow Cloud Services endpoint, which is represented by the value of the ACC_CNC variable in the one-line installer command. For information on the command and the parameters, see Install Agent Client Collector on Windows using ITOM Cloud Services and Perform a single-line Agent Client Collector installation on macOS by using ITOM Cloud Services.
The Agent sends registration request to the customer instance.
The Agent is registered in the customer instance and issued a certificate.
The Agent saves both the issued certificate and the public key used for verifying code signing signatures.
The Agent communicates with the customer instance through the ServiceNow Cloud Services by sending messages.
ServiceNow Cloud Services determine the correct customer instance to which agent messages must be sent.

Setup and Deployment

For ACC-VC, see KB1702432
For DEX, refer to the official DEX Documentation.

Special network considerations for ACC without MID

If you're using ACC without the MID server, you may need to configure ACC to use a proxy in order to connect to the ServiceNow Cloud. See KB1943452 for details.

Configuration

The agent can be used to monitor and discovery the host it is installed on. The actions performed by the agents are done via "Checks". An event can be created if a check fails and triggers an alert. The checks are grouped into policies. A policy will determine on what population of agents (Linux, windows, application, etc) to run the checks on as well as the frequency.

Agent Configuration

The main configuration files for the agent are the acc.yml and check-allow-list.yml. The check-allow-list.yml determines what commands the agent is allowed to run. Once a check is created click on related link "Generate allow-list content", next update the check-allow-list.yml file with the values generated so that the check is allowed to run on the agent.

Log parameters:

log-file	The log file location. Default = Windows: C:/ProgramData/ServiceNow/Agent-client-collector/log Linux: /var/log/servicnow/agent-client-collector/acc.log
log-file-and-stdout	Write logs to the stdout file. Default: false.
log-file-max-age	Maximum age, in days, of the log file before it is rotated out of the system memory. Default: 3.
log-file-max-backups	Maximum number of log files that can be stored before being rotated out of the system. Default: 3.
log-file-max-size	Maximum size, in MB, of the log file before it is rotated out of the system memory. Default: 10.
log-level	The log level to be measured by the logs. Available options are: Panic, Fatal, Error, Warn, Info, Debug. Default: Info. The specified log level represents the lowest level of events displayed in the log. For example, a user who specifies Error sees all Error events, as well as Fatal and Panic events.

Tables

https://yourInstance.service-now.com/sys_db_object_list.do?sysparm_query=nameLIKEsn_agent

Instance Scheduled Jobs:

https://yourInstance.service-now.com/sysauto_script_list.do?sysparm_query=sys_package%3Ddeb59787c317...

Agent Client Collector keepalive

Calculate Agent Feature Support Status

Refresh And Publish Monitoring Policies

Sync MID servers and Delete Drafts after policy import

Update Mid List For All Agents

Update Mid List For All New Agents

Update Processed Checks of Request

Checks

A check is a combination of a command and its configuration. The check is executed on the Agent Client Collector's servers. Checks are provided with the base system, and their commands execute scripts which provide monitoring data for your operating systems and applications. A check's default name indicates what is being monitored and measured, the entity, and the monitoring data. For example, a check named os.linux.check-system-cpu checks the CPU data on a Linux system. The identified command in the check runs on the monitored server, providing an output and status.

Create a Check Definition

Policies

Policies consist of the CIs monitored by the Agent Client Collector and the checks that run on those CIs. When creating a policy, you configure a filter which determines the specific CIs on which the checks are to run. For example, you can configure a policy to run checks on all Apache web servers. You can create new policies or edit the default policies, as needed.

Create a New Policy

Policy Lifecycle

Agent Client Collector Plugins / Assets

An Agent Client Collector plugin is a script or group of scripts which provides additional agent capabilities. For example, collecting more metrics, performing more checks, or generating events when an application queue size is 60% or 80% full. You associate a check with a plugin by creating a dependency between the check and the plugin. A plugin can have a dependency with several checks at a time; similarly, checks can depend on several plugins at a time. Plugins run on the same agent as the check.

You create Agent Client Collector plugins, as needed. Plugins are formatted as tar.gz files and run together with their associated check.

The plugins/assets can be seen on table sn_agent_asset, or by navigating to "Agent Client Collector > Configuration > ACC Plugins".

Create and Edit Plugins

Web Server

The Web Server is the "endpoint" to where the agents will connect. To view the web servers navigate to "Agent Client Collector > Deployment > MID Web Servers". There you will see the list of web servers running. Those are the servers to which the agents can connect, as well as the port.

Via the related links in this form you can stop, start, restart, test and update parameters.

MID Web Server Fail Over Configuration

The Agent's configuration file, acc.yml, determines to what MID Web Server it will connect. The configuration file for the agent allows for multiple Web Servers to be configured. The next MID Web Server configured in the acc.yml file will be used when communication cannot be established to a MID Web Server. The agents can also connect to a virtual IP address behind a load balancer.

Specify one or more MID server failover URLs in the acc.yml. The agent will communicate with the MID server using these URLs. This list is iterative; meaning the agent will try the first URL, if failed will move to the second URL and so on. If Auto-MID-Selection feature is on (the file mid.list.json is present), the agent will perform a connectivity test and will re-write the backend-url list. The order of the list will be based on ping time and then the number of agents already connected to the MID. This means that if the feature is enabled: the first MID server in the list should be the one with the lowest ping and the lowest number of agents.

To disable sending periodic MID Server updates from the ServiceNow instance to existing agents:
1. Navigate to System Properties > All Properties.
2. Set the sn_agent.enable_auto_mid_selection property to false.
To disable automatic MID Server selection for individual agents:
1. In the agent's acc.yml file, locate the enable_auto_mid_selection property.
2. Set the property value to false.

Lifecycles

How are the agents (sn_agent_cmdb_ci_agent and sn_agent_ci_extended_info) records created and updated?

Note: On the agent set acc.yml file set log-level = "debug", and set mid.log.level = debug on the MID server, to see detailed debug messages.

The agent uses the configuration from the acc.yml to connect to the MID Web Server
The agent sends keepalive events to ip:port/ws/events, when the agent first starts up you can see the keepalive loop starting:
```
{"component":"agent","level":"info","msg":"Starting keepalive loop","time":"2021-07-26T09:43:49-07:00"}
```

On the MID Server agent logs we can see the MID Server receiving the request and the reply:

07/26/21 10:29:53 (772) qtp540917385-156 DEBUG: (156) com.service_now.mid.webserver.jetty.WebServer - SERVER onFillable()
...
07/26/21 10:29:53 (773) qtp540917385-156 DEBUG: (156) com.service_now.mid.webserver.jetty.WebServer - onBinaryMessage(HeapByteBuffer@4079c03a[p=0,l=2293,c=2293,r=2293]={<<<keepalive\n{"timestamp":16...nse_required":"true"}}}>>>})
07/26/21 10:29:53 (791) qtp540917385-156 DEBUG: handleKeepalive: payload [{"timestamp":1627320593,"entity":{"entity_class":"agent","system":{"hostname":"......"}}}]

The MID Server sends an update to the instance (Update "Last refreshed" field or any other fields which need to be updated)
- On the instance:
  1. The Agent Client Collector API (/api/sn_agent/agents) receives the update sent by the MID server and updates the agent information
  2. The API is defined in /sys_ws_definition.do?sys_id=cf0d4208c3e3030039a3553a81d3ae9a
  3. The REST resource will update/create the agent accordingly

Finally the MID Server replies to the agent:

07/26/21 10:29:53 (791) qtp540917385-156 DEBUG: Publishing message [{ }] to client:WIN-IHKAN2LQJE1. remote: org.eclipse.jetty.websocket.jsr356.JsrAsyncRemote@3efeb957
07/26/21 10:29:53 (792) qtp540917385-156 DEBUG: (156) com.service_now.mid.webserver.jetty.WebServer - sendText("keep_alive_response
{ }")

On the agent we can see:

{"component":"agent","content_type":"application/json","level":"debug","msg":"message received","payload_size":3,"time":"2021-07-26T09:46:54-07:00","type":"keep_alive_response"}
{"component":"agent","level":"debug","msg":"keepalive response received from the backend. Setting the read deadline to 1627318314","time":"2021-07-26T09:46:54-07:00"}

Note: Scheduled job "Agent Client Collector keepalive" runs every minute to check for agents which the last_refreshed field has not been updated since the last keepalive and sets its statuses accordingly.

How are the policies and checks updated in the agent?

As part of the keep alive process, the MID Server checks for updates to the agent configuration (updates to checks, policies, etc)
The MID Server will reply to the keepalive and send additional information if there are updates to be sent to the agent. In the MID Server agent logs we can see, for example:
```
07/26/21 12:32:53 (776) qtp540917385-157 DEBUG: Publishing config {"checksum":"826562631", 
 "check_requests":[{
  ...
}]} to client: name {
...
}, agent_id 5d06b30ec6c4ce0b
```

How are the policies and checks updated to the MID Web Server?

Job "Refresh And Publish Monitoring Policies" (sysauto_script.do?sys_id=126cee2bc3b513002a6f741e81d3ae87) runs every minute
The job calls MonitoringConfig.syncPoliciesToMid(); which checks if we need to send an update to the MID Servers
If an update must be sent to the MID server, create an ecc_queue output record with topic "MonitoringProbe" and source = "config_publish"
The MID server process the output and updates the agent policies

How are assets downloaded to the agent?

The assets files are synchronized to the MID servers via the FileSyncer
- The FileSyncer is a process/thread that keeps MID Server files synchronized to instance files
- The assets can be seen in the MID server folder:
```
%INSTALL_FOLDER%/agent/static/acc_plugin
```
The keepalive process which keeps the agent record on the instance up to date, as well as policy/checks on the agent client, also contain an "assets" section

The assets section of the payload tells the agent what assets need to be downloaded, for example:

  "assets" : [ {
    "sha512" : "c5149676070a227ab75a6c2979a568404e6710c9c8d57766d85a7c759504b833788ad133b99caa8d53cbd134fe42817b84eef1880b41d29a8e0e2f51fe8d5c73",
    "url" : "{{MID_URL}}/static/acc_plugin/windows/all/all/all/acc-visibility-modules-windows.tar.gz",
    "metadata" : {
      "name" : "acc-visibility-modules-windows",
      "namespace" : "default"
    }
  }

The agent downloads the assets from IP:PORT/static
In the agent, the files are downloaded to its "cache" folder
- Windows: C:/ProgramData/ServiceNow/Agent-client-collector/cache
- Linux: /var/log/servicnow/agent-client-collector/cache

This file synching uses a pre-existing MID Server sync feature. Detailed information on how this works in general, known issues, and debugging tips can be found in:
KB0852276 How MID Server File Synchronisation works, to help when Troubleshooting

How are results sent from the agent to the instance?

The checks will run as configured in the policies

The agent sends the check result to the MID Server using the WebSocket connection

On the agent acc.log we can see the result being run and sent, for example:

{"component":"agent","level":"info","msg":"Running check, name: policy: Windows OS Metrics, check:os.windows.metrics-system-cpu-load ...
{"component":"agent","level":"debug","msg":"{winchecks metric-windows-cpu-load [AGENT_DIR=C:\\Program Files\\ServiceNow\\agent-client...
{"component":"agent","content_type":"application/json","level":"info","msg":"sending message","payload":"{\"timestamp\":1628002490,\"...

On the MID Server we can see the result received

DEBUG: handle result from client [WIN-IHKAN2LQJE1] agent_id [5d06b30ec6c4ce0b], check result is [{"timestamp":1628004950,"check":{"co...
DEBUG: MID Script Include cache hit for name=MonitorResultParser

On the MID Server the MID script include MonitorResultParser checks if the check result has the mid_script field populated, this will be configured on table sn_agent_check_type
Is the mid_script field populated?
- Yes: Run the mid_script to handle the result
  - Depending on the mid_script the result may, for example:
    - Sent back to the ecc_queue as an input
    - Sent back as an event
    - Sent back as a metric
- No: Send the result to the ecc_queue
Once the result is in the instance, the sn_agent_check_type instance script will process the result

Troubleshooting