Last time, we saw how statistical sampling was used in an election poll: among the population size of 133 million likely voters, a sample size of 786 was used to show the percentage of votes for each candidate, with the margin of error of 3.5% at 95% confidence level. We saw how a relatively small number of samples could provide an insight to a very large population size. We also saw that the insight was only an estimate given as a range of possible values, called confidence interval, and the results should be carefully interpreted for their statistical significance.
Election polls, as well as other national polls on various topics, have a unique characteristic in that their population size is huge, typically in the tens or hundreds of millions. For such large population sizes, the sample size only depends on the margin of error (and confidence level) and the math becomes somewhat simpler, as we saw last time. When we're working with ServiceNow, we typically deal with records in the thousands to hundreds of thousands, and perhaps millions in some cases. This alters the math somewhat, which we'll see why and how later, while all the statistical concepts still remain the same and useful.
CMDB DATA QUALITY
The CMDB plays an important role in ITSM and beyond, as it's the central repository of what we have and how they interact. Yet it's often a tremendous challenge to maintain a reliable CMDB due to data quality issues, whether the data are auto discovered or manually populated. This poses a hindrance to the effective use of the CMDB and may lead to the viscous cycle that ends with the death of the CMDB; the data are unreliable, so people stop using them, so people stop updating them, so the data become stale, and it keeps getting worse. The CMDB needs tender loving care for it to thrive and maintain good health.
CMDB data quality has two sides: accuracy and completeness. These apply to both CI attributes as well as relationships. In some organizations, CMDB data are periodically reviewed, and certified, from operational or compliance reasons. Typically, there are two distinct steps that ensure data quality:
- Data certification
- Independent review/audit
Data certification is typically performed by CI owners periodically (for example, using ServiceNow's Data Certification application) or built into Change Management / auto discovery processes. If the number of CIs is small, all CIs may be included. However, as the number of CIs increases, the effort may become too time consuming for CI owners. In that case, a risk-based approach may be taken to adjust the scope, periodicity, and methodology. For critical CIs, including those subject to SOX (financial) or GxP (life sciences) regulations, continue certifying all CIs. For non-critical CIs, use statistical sampling with a predefined threshold; for example, certify if 95 ± 5% of the samples pass (that is, at least 90% pass), otherwise take corrective actions.
Independent review/audit can almost always utilize statistical sampling (for example, using ServiceNow's GRC application). As we just did above, a predefined threshold can be used to assess the effectiveness of data certification; a result below the threshold may lead to a revocation of data certification and other corrective/preventive actions.
When performing these steps, the population can only be defined from the known CIs; if there are missing CIs or relationships, they won't be counted. Therefore, any missing CIs should be identified and added before drawing samples. Also, using only pass/fail for evaluation keeps the statistics and reporting simple. For example, for a CI, even if only one attribute is incorrect out of ten, it would still get a fail grade.
Next time, we'll do some number crunching and see how this works in practice, using a fictitious CMDB.
Please feel free to connect, follow, post feedback / questions / comments, share, like, bookmark, endorse.
John Chun, PhD PMP 
1 Comment
