Last time, we saw how statistical sampling techniques can be used for data certification, using a fictitious CMDB as an example. We also saw how sample size is calculated and how results are interpreted. A practical use case was introduced for conducting a "quick" CMDB assessment where a small sample of only 383 CI records gave an insight to a CMDB with 100,000 CIs. This time, let's take a look at how statistical sampling may be used in ServiceNow.
GOVERNANCE, RISK, AND COMPLIANCE (GRC)
The GRC application in ServiceNow (available as a plugin) is used to manage an organization's objectives and various regulatory requirements, if applicable. Among the tools it provides, Control Test Definition (or Indicator in Helsinki) has an ability to specify a sample size for controls testing; when a control test is executed, a random sample is generated for review. Let's take a look at how this actually works in ServiceNow and discuss limitations and workarounds.
NOTE: The GRC application has gone thorough a major redesign with the Helsinki release, so there are quite a few differences from the previous releases in terminology and how the application is organized. The sampling feature being described here, however, works similarly across Helsinki and previous releases.
In pre-Helsinki releases, you can get to Control Test Definitions by navigating to
IT GRC > Administration > Control Test Definitions or
GRC > Administration > Control Test Definitions (requires "grc_test_definition_admin" role).
In Helsinki, navigate to
Policy and Compliance > Indicators > Indicators (requires "sn_compliance.user" role).
Based on the "quick" CMDB assessment use case from Part 3, I created a new definition shown below in the Geneva release:
Name: CI attributes are correct
Control: Assign ownership of the configuration item to the appropriate owner and set to proper status.
Collect supporting data: checked
Condition type: Basic
Data purpose: Support test execution
Table: Configuration Item [cmdb_ci]
Fields: Owned by, Status
Sample size: 383 (manually entered)
Condition type must be set to "Basic" and Data purpose set to "Support test execution" to use sampling so the Sample size field becomes visible. In this simple example, we're reviewing the entire CMDB, so set Table to "Configuration Item [cmdb_ci]" but don't set Control test condition (filter) to anything. Since we're reviewing only the CI ownership and status, set Fields to "Owned by" and "Status" (in real cases, there may be more fields being reviewed). Finally, set Sample size to 383, which is what we used in Part 3 for the CMDB with 100,000 CIs (this is a manual entry field, so any value can be entered; setting it to 0 returns all records).
When Execute Now is clicked, a new Control Test Instance is created in the Related List section at the bottom of the form (reload the form to see it). The newly created Control Test contains the sample of records to be reviewed; any discrepancies can trigger Remediation Tasks.
TESTING FOR APPLICATIONS OR OTHER CI TYPES
If testing for applications only (or any other CI types), create a new Control Test Definition (or Indicator) and set Table to "Application" and adjust the Sample size based on the Application population size. The population size can be further reduced by using Control test condition (filter).
LIMITATIONS AND WORKAROUNDS
When using Control Test Definition, population size is determined by the result set returned from Table and Control test condition. Unfortunately, Control Test Definition doesn't provide population size or an automatic way to calculate sample size. Also, population size tends to fluctuate over time since the CMDB can be dynamic. Therefore, for each test, population size has to be determined by using a list view for the CMDB outside of Control Test Definition, with the same Table and Control test condition to filter and return the same result set. Then the population size can be used to calculate sample size.
If this becomes cumbersome, an over-estimated sample size can be used that doesn't need to be revised for every test. As we have seen, sample size changes little when population size changes dramatically; in Part 3, the sample size barely changed from 370 to 383 when the population size changed from 10,000 to 100,000 (tenfold). If the population size is in tens of thousands today and it's not expected to exceed 100,000, then a fixed sample size of 383 can be used for all future testing. In effect, this over-estimated sample size will result in a slightly smaller margin of error, below 5%; in practice, this may be insignificant and won't alter the conclusions.
An automatic sample-size calculator may be an enhancement we can hope for in a future release.
Next time, we'll look at other related features from ServiceNow.
Please feel free to connect, follow, post feedback / questions / comments, share, like, bookmark, endorse.
John Chun, PhD PMP 
1 Comment
