Agent Client Collector for Visibility(ACC-V) for Software asset management of end user devices

[DD786]

Contents

Purpose

Pre-requisites

What is it

Part 1- Discovery of EUCs using ACC-V

Part 2 – Consuming the output for SAM

Software Usages/Metering

Sam Reconciliation & Optimization

Important Tables & Jobs

Useful Properties

FAQs

Useful Links

Purpose:

This write-up aims to capture details on how agent client collector for visibility can help in software asset management of end user computers(EUC).

Pre-requisites:

Below plugins has to be activated:

1. Discovery and Service Mapping Patterns (sn_itom_pattern).
   1. At least Discovery (com.snc.discovery) should be activated.
2. MID server to connect to the network
3. Agent Client Collector for Visibility (sn_acc_visibility)
4. Software Asset Management Professional(com.snc.samp) and publisher packs(as per the need)

What is it:

Agent client collector, as the name suggests, is basically ServiceNow agent installed on your machine which then helps to discover the software installed and their usages in your machine.

To establish end to end license positioning of software for EUCs, the process goes through two parts:

1. [ITOM] Discovery of EUCs, software installs and usages from the source
2. [SAM] Leverage the output to obtain license positioning and license harvesting

Part 1- Discovery of EUCs using ACC-V:

Refer Exploring Agent Client Collector for Visibility

&&

Agent Client Collector Visibility

Part 2 – Consuming the output for SAM:

When the configuration is successful and the agent starts transmitting information to ServiceNow instance, below events occurs:

1. The EUCs gets registered in Computer[cmdb_ci_computer] table with their details.
2. Each EUCs gets reported with software installed in them.
3. Software Usages gets reported in SAM table.

When the set up is correctly done, you should see below tables getting populated immediately:

• Table: Agent Client Collectors [sn_agent_cmdb_ci_agent]

This table holds all the devices where agents are running successfully, once the source machine(EUCs) is connected to the network. Column ‘Host’ will have the actual machine as registered in the CMDB.

• Table: Software Installation [cmdb_sam_sw_install]

This table holds the software installations detected from the source. You need to navigate to the host and access the software installations tab from the related list of the host.

• Table: Software Usage [samp_sw_usage]

This table holds usages of all the software from the source, for which reclamation rule has been defined.

Software Usages/Metering:

In SAM, ACC_V can fetch two types of usages of software from the source. Usage based on last used date and total usage durations.

Basic Metering (Last Used):

Basic metering refers to getting software usages from the source machine based on Last used data.

Detailed explanation is kept in the doc : https://docs.servicenow.com/bundle/xanadu-it-operations-management/page/product/agent-client-collect...

Some key points to consider:

1. Osquery is not needed from implementation/configuration view point.
2. ‘SAM discovery’ policy fetches the needed usage with last used time stamp.
3. In Windows end points, this data is taken from UserAssist table in the registry.
4. disable_sam_reclamation_rules_for_licensable_softwares property controls if reclamation rule should be used to create the usage record in the software usage table. Click here for more information.

Total Usage Metric:

Dependency:  Other than functionalities, plugins mentioned in the above pre-requisites section, total usage fetch for software from the source demands Osquery installation in the source.

 

High-level Architecture:

Explanation:

***ACC Installation without Osquery enables ACC to report the software installations with Last used date. This is used for Basic metering feature of SAM. Basically, if a reclamation rule is created where software has to be reclaimed based on “Last Used date”, this data from basic metering is used.

In order to get total usage of software from endpoints, Osquery package has to be installed on top of existing ACC installations.

Osquery package can be downloaded from https://osquery.io/downloads/official

For Windows machines, “.msi” file should be downloaded.

And then can be installed manually or using package deployment applications like SCCM, Intune etc.

Post successful installation, one should see below folders in the endpoint under program files:

One has to ensure osquery.conf file is created with this content (Attachment in KB1005005). 

And then update the file as mentioned here Configure Osqueryd schedule for SAM total usage metrics

Then osquery.flags Flags file has to be updated as mentioned here Configure Osqueryd logs for SAM total usage metrics. A sample of how the flag file should look like can be found here(Attachment in KB1005005). 

Osquery installation creates a ‘marker.json’ file in below path <userprofile>\\AppData\\Local\\AgentClientCollector\\SAM

After which, one should start the osqueryd service(daemon) in the machine. Restart if already started.

This daemon updates this marker.json file periodically and stores incremental/delta data.

This ‘marker.json’ file stores important data like the process ID, usage times and user related data. File should look like this:

Also a snapshot of this usage data is kept for reference here in ‘osqueryd.snapshots’ file under below path:

The query that the system fires to generate above output of marker.json file is defined under “sam-metering.conf” here:

Query defined in this configuration file:

{

  "platform": "windows",

  "queries": {

    "sam_process_info": {

      "query": "select name ,pid, elapsed_time, start_time, user_time, system_time, username from processes p JOIN users u ON u.uid = p.uid where p.elapsed_time != -1 ;",

      "snapshot" : true,

      "interval": 300

    }

  }

}

Now, when ACC policy : ‘SAM discovery’ gets executed. The check ‘sam advanced metering’ gets triggered which then invokes below ruby files as mentioned in the Command field of the check:

endpoint_discovery.rb --compact --select=data_collection,enhanced_inventory,installed_software,sam_advanced

Whereas all the other ruby scripts like data_collection, enhanced_inventory, installed_software are used for collecting cmdb info of the endpoint plus software installed on it, sam_advanced script has the logic to read the marker.json file and generate the output in needed format that gets appended with the output of the above scripts and then gets communicated to ECC queue via the MID Server.

In the endpoint, total usage related ruby scripts are stored here:

In the ECC queue, payload is received from the agent in below format:

SOME XML TAGS

"client" : "[DEVICE_ID]",

  "agent_id" : "d30xhas2321",

  "check" : {

    "command" : "endpoint_discovery.rb --compact --select=data_collection,enhanced_inventory,installed_software,sam_advanced",

    "name" : "policy: SAM discovery , check:Software installations and usage metrics",&#13;

    "interval" : 3600,

    "timeout" : 600,

    "output" : "{

         "basic_inventory":{"some data..."},

         "serial_numbers":[{"some data..."}],

         "enhanced_inventory":{"some data..."},

         "installed_software":["some data..."],

         "edition_info":[],

         "os_package":{"some data..."},

         "caption":"some data...",      

         "sam_advanced":[{\"user_name\":\"\",\"processes\":[{\"total_elapsed_time\":354137,\"name\":\"AppVStreamingUX.exe\",\"total_usage_count\":1,\"total_user_time\":1703,\"total_system_time\":2187,\"pid\":\"11896\",\"start_time\":\"1694676511\",\"username\":\"\",\"ad_domain\":\"global.hvwan.net\"},{\"total_elapsed_time\":354136,\"name\":\"sihost.exe\",\"total_usage_count\":1,\"total_user_time\":40218,\"total_system_time\":52703,\"pid\":\"4976\",\"start_time\":\"1694676512\",\"username\":\"\",\"ad_domain\":\"global.hvwan.net\"},..."}],

                  "unixTime":1695030648,

                  "error_msg":\"\"}

    "status" : "0",

    "ci_id" : "78f35565db3fdb00f3bdf5dcaf96196c",

    "payloadId" : "9da22abf-84bf-4a52-8048-69385456f6f1",

    "check_instance_id" : "470286d41b593510fc61a75ee54bcb0a",

    "check_type_id" : "5e34d4ee87450510829cffb2debb351c",

    "params_last_updated" : "2023-09-15 08:19:15",

    "params_background" : "false",

    "disable_token_replacement" : "false",

    "exec_mode" : "execv"

  } .....

  SOME CLOSING XMLS TAGS

Part of the payload above “sam_advanced” key is used for CMDB and SAM basic metering purpose.

All the data under “sam_advanced” key is then consumed for total usage loading in ServiceNow table.

Business rule “AgentNowResponseProcessor” fires when an input is received from the agent/endpoint in the ecc queue. This BR invokes Script Include “new AgentNowHandler().processEccRecord()” which actually processes the payload.

This BR then loads the data into SAM Software Usage Stagings [sn_acc_vis_content_sam_software_usage_staging ] table.

In order to enter data into this staging table, the logic looks into :

1. Existence of a reclamation rule for the product with reclamation type= total usage time.
2. Processors defined in this rule.
3. Discovery of the endpoint(EUC).
4. Discovery of the same software in the endpoint/agent.
5. Discovered software model should be normalized.                                
6. User that is sent in the SAM payload should match with the userId(user_name field) from the users table (sys_user) in the instance

Once above checks are passed, this staging table gets loaded with software and their usages.

Like below:

                             

From here, a monthly SAM job i.e., “SAM - Update Software Total Usage Metric” loads the data to Software usage table (“cmdb_sam_sw_usage”).

And then reclamation candidates get generated based on the reclamation rules .

Sam Reconciliation & Optimization:                                                                 

With the endpoints getting discovered properly i.e., with all the attributes and software installs, SAM reconciliation can progress.

SAM admin has to load the entitlement as per the contract and once the reconciliation completes, compliance position gets positioned in the workspace.

Below is how MS Office 2019 Standard getting positioned when discovered via ACC-V:

• Machine - 22c…71 got discovered via ACC-V and MS Office 2019 standard installs are found in the same machine.                                                                                      

• Post reconciliation, this is what the workspace displays:

• Also, optimization logic triggers and if reclamation rules are created, removal candidates get generated accordingly.
• Below we could see all the removal candidates generated due to reclamation rule MF Office – 90 Days. This rule is configured to reclaim software which are not used from last 90 days.

• Machine – 22c…71 is also stamped for removal as MS Office was last seen in this machine before 90 days.

Important Tables & Jobs:

• Computer [cmdb_ci_computer] : End point computers/devices will be loaded here
• Agent Client Collectors [sn_agent_cmdb_ci_agent] : Corresponding agents of the host swill displayed here
• Software Installation [cmdb_sam_sw_install] : Software installed on each hosts
• Software Usage [samp_sw_usage] : Software usage information from the host
• SAM Software Usage Stagings [sn_acc_vis_content_sam_software_usage_staging]  :Staging table to hold software usage information.
• ECC Queue [ecc_queue]  :Raw output from source machine will be loaded in this queue.
• SAM - Update Software Total Usage Metric  :Job to load data from Software usage staging table to Software usage table

Useful Properties:

• sn_acc_vis_content.assigned_to_user_order[reference found here ] :  The order of username sources defined here is used to define the priority based on which the Assigned To user is set for a computer CI.
• set_assigned_to [reference found here ]  : If set to true Assigned To attribute of computer CI will be overridden by ACC-V.
• column_name_for_user_mapping [reference found here ] : This property is utilized to associate a user (assigned_to) with an agent CI and the SAM usage according to the Active Directory (AD) domain. The value of this property determines which field in the sys_user table will be used to validate the username and domain, ensuring the correct mapping to the appropriate CI.
• disable_sam_reclamation_rules_for_licensable_softwares [reference found here ] : Controls whether reclamation rules should be used for software usage load or not.
  • How does this property behave?
    •  If 'false', store SAM usage according to defined reclamation rules.
    •  If 'true', store usage according to defined reclamation rules + store usage for all the licensable softwares (for which no reclamation rule is defined).
  • network_adapter_exclusion_list [reference found here ] : list of regular expressions for the names and IP addresses which will be excluded in the Newtork Adapter and IP Address tables.
  • persist_os_user_type [reference found here ] : OS user data to be persisted through ACC-V discovery.
  • persist_sam_usage_metrics [reference found here ] : If true, then persist SAM related information

FAQs:

• ‘Software Installed Policy’ is not getting picked for my new device : Mostly this issue occurs when the device class(CMDB CI class) is different than the classes defined in the Filter of the policy. Or some other condition in the Filter which will render this device out of scope.
•  Can we adjust the schedule of the ‘SAM - Update Software Total Usage Metric’ job to run daily, weekly, or at a custom interval, so that the total usage data is collected more frequently? : Yes you can. But then 2 factors we should consider:
  • We should think about why do we need to. This job is not only about total usage data collection for software in the source but also taking a decision on optimization on the collected data. And hardly, we have seen any optimization decision is taken by any organization without evaluating the software usage pattern of at least one month.
  • The script include involved have condition to collect data from the staging table which are of one month old only. So even if you reschedule the job to closer frequency, system will only populate data in software usage table for usages older than a month.

• Can I run a policy before 24 hrs(default duration) : In a policy, under scheduling section, one can schedule the policy run time. If this schedule is pushed to a closer frequency, say daily twice or thrice etc, consider the impact in the platform. In this case, more the number of agents discovered, more transactions will be processed.
• I changed the policy schedule to run daily three times still it is actually running daily once : Check if the override fields of the corresponding ‘check instances’ are not set. Override interval/Override cron expressions field data takes precedence over schedule of the policy, if not empty.
• I do not see software usage table getting populated with records whereas installation records are coming after ACC-V is set up : Well it can be due to below reasons:
  • ‘persist_sam_usage_metrics’ property is set to true or not.
  • Reclamation rule is set up
  • For Total Usage Metering:
    •  If usage records are getting populated in the staging table [sn_acc_vis_content_sam_software_usage_staging]
    • If “SAM - Update Software Total Usage Metric” scheduled job is active and scheduled.
    • Reclamation rules have corresponding ‘Product Process’ identified.
    • More probable scenarios discussed here KB1005005

Useful Links:

• SAM Metering Setup with ACC: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1642676
• Agent Client Collector: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0815247
• Agent Client Collector Visibility: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0966481
• ITOM Agent Client Collector documentation material: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1122613
• Osquery : https://osquery.io/downloads/official
• ACC-V SAM - Update Software Total Usage Metric: https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB1005005
• Agent client collector: https://docs.servicenow.com/bundle/xanadu-it-operations-management/page/product/agent-client-collect...
• Using push-based Discovery and horizontal IP-based Discovery together : https://docs.servicenow.com/bundle/xanadu-it-operations-management/page/product/agent-client-collect...
• Using push-based Discovery and SAM together: https://docs.servicenow.com/bundle/xanadu-it-operations-management/page/product/agent-client-collect...  

[Community Alums]

Thanks for sharing

[SamirNyra]

Greetings, 

This is a wonderful blog and i am now hopeful that you will be able to help me with below. 

We need some help on  step 1 SAM total usage metrics flowchart  flow chart - Using push-based Discovery and SAM together

Which ECC Input Queue should we be looking at ?

We are getting ECC inputs for Checks – Enhanced discovery and Software installations and usage.

However the SAM usage staging table is still empyty 😞 .. I have checked ACC logs and all checks seems to be running and all system properties are set as it should be .

Any help is greatly appreciated.

[DD786]

@SamirNyra 
Is it an EUC?
If yes, can you see the payload containing data under '"sam_advanced":[]' key.
Refer the article for what to expect under this key.
If yes, Is reclamation rule with total usage type for that product, created .
If yes, has the reclamation rule tagged with corresponding product process.
If yes, is this product process available under content of "sam_advanced":[]'

[SamirNyra]

Hello DD7786, 

Apologies to address with your user id as don't know your real name.

This is EUC and the payload we have been getting for SAM advanced is usually empty if running in the back ground and when i don't run in the background i still can't see any meaningful payload. 

However i have now seen that there is some data coming to the SAM Software Usage Staging Table as shown below . Unfortunately no more data is coming other than the 11 records created . Moreover not all Installed Software is picked yet. A user has 501 installed software however only 5 is being shown here. 

However it is not getting pushed to the Software Usage Table 

I promise to deep dive and read your blog again for finer print. But if you could be kind enough to clarify couple of key points.

1) Reclamation rule - we have a property to suggest we can stage the data even if they are no defined. . I have selected the value as True. 

What i am not clear is if this property is only applicable to - SAM Software Usage Staging Table and not for Software Usage Table or is it applicable for both ?  

The other observation i have is that software usage is also not present for normalized software as well.

Apologies for long reply but any help will be vastly appreciated.

[SamirNyra]

<results probe_time="0" result_code="0">

<result>

<output>[ { "client" : "WPPIT-4987", "agent_id" : "c82425bc15073a47", "check" : { "command" : "sam_processor.rb", "name" : "policy: Combined Policy test with no filter for Dev , check:SAM background log check", "interval" : 300, "timeout" : 2300, "output" : null, "status" : "0", "ci_id" : "fbc24f2d9377b1105bd6ff5d6cba1046", "payloadId" : "0de6fa28-4433-418f-b178-ad6af09fd40f", "check_instance_id" : "e1a0482997fd9254ffde3756f053af0d", "check_type_id" : "75fb38031c464ab0b45efcfc14ae884e", "params_last_updated" : "2024-11-14 09:31:25", "params_background" : "false", "disable_token_replacement" : "false", "exec_mode" : "execv" }, "timestamp" : 1731581671 } ]</output>

</result>

<parameters>

<parameter name="name" value="on_demand_request"/>

<parameter name="topic" value="MonitoringProbe"/>

<parameter name="source" value="on_demand_request"/>

<parameter name="priority" value="2"/>

</parameters>

</results>

[Stev Williams1]

Great guide thanks for putting it together.

However for some reason I cannot get osquery to generate the marker.json file and i have no idea why.

Can someone confirm when adding - 

"packs": {
      "sam-metering": "C:\\ProgramData\\ServiceNow\\agent-client-collector\\cache\\acc-visibility-modules\\bin\\sam-metering.conf"
  }

to the config file its as simple as adding the line here a the bottom the "packs" in the config file?

  "packs": {
    // "osquery-monitoring": "/opt/osquery/share/osquery/packs/osquery-monitoring.conf",
    // "incident-response": "/opt/osquery/share/osquery/packs/incident-response.conf",
    // "it-compliance": "/opt/osquery/share/osquery/packs/it-compliance.conf",
    // "osx-attacks": "/var/osquery/packs/osx-attacks.conf",
    // "vuln-management": "/opt/osquery/share/osquery/packs/vuln-management.conf",
    // "hardware-monitoring": "/opt/osquery/share/osquery/packs/hardware-monitoring.conf",
    // "ossec-rootkit": "/opt/osquery/share/osquery/packs/ossec-rootkit.conf",
    // "windows-hardening": "C:\\Program Files\\osquery\\packs\\windows-hardening.conf",
    // "windows-attacks": "C:\\Program Files\\osquery\\packs\\windows-attacks.conf"
 "sam-metering": "C:\\ProgramData\\ServiceNow\\agent-client-collector\\cache\\acc-visibility-modules\\bin\\sam-metering.conf"
  },

[SamirNyra]

"packs": {
// "osquery-monitoring": "/opt/osquery/share/osquery/packs/osquery-monitoring.conf",
// "incident-response": "/opt/osquery/share/osquery/packs/incident-response.conf",
// "it-compliance": "/opt/osquery/share/osquery/packs/it-compliance.conf",
// "osx-attacks": "/var/osquery/packs/osx-attacks.conf",
// "vuln-management": "/opt/osquery/share/osquery/packs/vuln-management.conf",
// "hardware-monitoring": "/opt/osquery/share/osquery/packs/hardware-monitoring.conf",
// "ossec-rootkit": "/opt/osquery/share/osquery/packs/ossec-rootkit.conf",
// "windows-hardening": "C:\\Program Files\\osquery\\packs\\windows-hardening.conf",
// "windows-attacks": "C:\\Program Files\\osquery\\packs\\windows-attacks.conf"

"sam-metering": "C:\\ProgramData\\ServiceNow\\agent-client-collector\\cache\\acc-visibility-modules\\bin\\sam-metering.conf"
},

[SamirNyra]

Yes adding a line did work for me however i am not sure what is the role of this pack as heavy  lifting is done by the osquery !!  I even checked binary but can't find any thing.. we have to add because documentation say so !!!

[Tone1]

Hello everyone,

Great guide overall! I wanted to share an additional tip regarding osquery metering.

Running the query

SELECT name, pid, elapsed_time, start_time, user_time, system_time, username FROM processes

every 5 minutes can cause significant overhead, especially on systems with multiple users, such as Microsoft RDS environments.

A more efficient approach is to leverage the Event Tracing for Windows (ETW) integration in osquery, which is event-driven rather than polling-based.

For more details, check out this discussion on ETW and osquery:
https://github.com/osquery/osquery/issues/7826

With ETW, every process start and stop event can be logged directly into osquery's RocksDB. Then, an off-hours job can aggregate the data and push it into the ServiceNow instance.

By using the right decorators and osquery’s built-in math functions, it should be possible to extract the required information in the exact format expected by sam_processor.rb.

This could help improve efficiency for those monitoring process activity with osquery in the future.