Application Scoring and Application Assessment in Enterprise Architecture

[Mathew Hillyard]

This blog attempts to demystify how application scoring and particularly application assessment functions with Enterprise Architecture (EA), previously known as Application Portfolio Management (APM), including some of the issues and limitations around how applications are assessed, and some tips on how to configure EA appropriately in order to get the desired results.

This blog post is applicable to instances on Yokohama Patch 3 with EA v1.0.0 onwards.

Table of Contents

• Application Scoring Architecture
• Creating Assessments
• Application Scoring
• EA Workspace Visualizations
• Summary
• Key Takeaways
• Acknowledgements and Links

Application Scoring Architecture

Application Scoring Profile

Business Applications are scored when the Application scoring profile [apm_application_profile] field is populated with an Application Scoring Profile. This field is in the Enterprise Architecture application scope, although the Business Application table is in the Global scope – so be careful with ACLs, UI Policies etc. as they may not work or even be possible.

Application scoring profiles are available from within EA Workspace > Setup > Scoring Profiles > All. In the baseline there are two profiles available:

• Business Capability Scoring Profile – for assessing Business Capabilities.
• Default Application Profile – for assessing Business Applications

If you install the Cloud Assessment application from the Store, a third profile will be created - Cloud Readiness Scoring Profile – for cloud readiness assessment. This also adds an Assessment Data tab to the Business Application form (Business Application view only).

Application Profile

This resembles a many-to-many table and enables you to associate one or more Application Profile Indicators to an Application Scoring Profile. This is where you should set the Weightage for a given Indicator, which is summed to create the overall Application Score. This allows Application Profile Indicators to have a different Weightage value for each Application Scoring Profile they belong to. Records are stored in the Application Profile [apm_application_profile] table.

Application Profile Indicator

Each Application Scoring Profile is made up of one or more Profile Indicator [apm_metric] records. A Profile Indicator can generate scores based on one of five possible data sources:

1. Performance Analytics – this refers to either the legacy Performance Analytics, or current Platform Analytics (PA) Indicators. select a PA Indicator and choose a default breakdown to return scores for the target record. The breakdown should be the same table as the target table on the Profile Indicator, so for the Default Application Profile, the breakdown should be on Business Application. The Indicator should be related to a running Collector Job.
2. Assessments – reference an Assessment Metric Type and Assessment Metric Category to return scores from completed assessments.
3. Custom script – write a script to interrogate a table and return a list of record Sys Ids that correspond to the Profile Indicator’s CI Class (e.g. Business Application), and each record’s corresponding “weight” (score).
4. Query Condition – use a field value on the Profile Indicator’s Query table (e.g. Business Application) to aggregate and return values using standard mathematical aggregation (SUM, COUNT, AVERAGE, MIN, MAX) to obtain a range of values, and optionally consolidate the scores together.
5. Indicators – this acts as a parent Indicator and allows you to group together and consolidate values from other (child) Profile Indicators via the Indicator Dependencies related list. Note that both parent and children appear in the Application Scoring Profile’s Profile Indicators related list. However, only the Parent Indicator is considered when scoring as its Weightage is an aggregate of its child Indicators.

Other Profile Indicator fields

• Unit – unit of measurement to be used for the Profile Indicator – for example the Total change hours Profile Indicator counts the number of hours between Change Requests being opened and closed, so the unit is set to Hours.
• Frequency
  • Month
  • Quarter
  • Year
• Target maximum, Target minimum and Consider Absolute Values are available for all data sources except Assessments and are related to normalisation of application scores:
  • If the Target maximum and Target minimum are not set, then the maximum value within the range of applications is taken as the target maximum value. Similarly, the minimum value within the range of applications is taken as the target minimum value.
  • If the  Target maximum  and  Target minimum  are set and the Consider Absolute Values check box is selected, the entered values are considered.
  • If the Target maximum  and  Target minimum  are set and the Consider Absolute Values check box is cleared, the values are considered based on the following intelligent logic:
    • Target maximum = Minimum value of (Target maximum value defined in the Indicator [apm_metric] table, Maximum value of Application Weights for the fiscal period)
• CI Class – as this is EA it is restricted to Business Application or Business Capability.
• Direction (Maximise or Minimise) – so the Indicator can calculate the application score appropriately – i.e., with Maximise, a higher value is better, whereas with Minimise, a lower value is preferred.

The Baseline Profile Indicators

There are a total of 28 Profile Indicators shipped with EA:

Default Application Profile

Application Operational Metrics

Parent Indicator for child indicators Number of Changes via Service, Number of Incidents via Service, and Number of Problems via Service.

Application's Incident Count

Custom Script Indicator counts the number of Incidents between the defined start and end dates (based on the indicator frequency) for the value in the “Business application” field on the Incident record. This is deprecated and no longer in use as of CSDM. It is a redundant Profile Indicator as the Number of Incidents via Service retrieves the same information but correctly from the Impacted Business Application [task_cmdb_ci_business_app] table. It’s best to remove this Indicator from any Application Scoring Profiles you use and deactivate it.

Application's Instance- Incident Count

Custom Script Indicator that calculates the number of Incidents between the defined start and end dates, this time by looking for Incidents, retrieving the value in the Service [business_service] field, then locating a CI Relationship where the Parent’s Class is Business Application and the Child is the Service. This is not the correct CSDM relationship as a Service is related to a Business Application via one or more Business Capabilities or via child Service Offerings, then related Application Services, and finally related Business Applications; furthermore, it would only work if you opt to populate the Service field on Incident with an Application Service (now re-labelled as Service Instance), which is not a method I would recommend – I would align with the leading practice method instead. It is also a duplicate of the Number of Incidents via Service Indicator. Again, it’s best just to remove this Indicator from any Application Scoring Profiles you use and deactivate it.

Business Value

Assessment Indicator with a single Metric Category and Metric related to business value.

CSAT

Assessment Indicator with a single Metric Category and Metric related to customer satisfaction.

Functional Fit

Assessment Indicator with a single Metric Category and Metric related to the functional fit of the application.

Number of Changes via Service

PA Indicator which counts the number of Change Requests within the given review period that reference a given Business Application in the Impacted Business Applications [task_cmdb_ci_business_app] table (which is the Impacted Business Applications related list on the Change Request). This requires CSDM services to be configured, the form populated with at least a CI and the Change Property “Populate the Business Application related list for change requests” to be ticked in order to generate data.

Number of Incidents via Service

PA Indicator which counts the number of Incidents within the given review period that reference a given Business Application in the “Impacted Business Applications” related list. This requires CSDM services to be configured, the form populated with at least a CI and the Incident Property “Populate the Business Application related list for incidents” to be ticked in order to generate data.

Number of Problems via Service

PA Indicator which counts the number of Problems within the given review period that reference a given Business Application. This indicator will work as intended, however in the baseline platform there is no automation on the Problem table to calculate or populate Impacted Business Applications (or any of the other Problem “impacted” tables either for that matter). So, in order to get scores for this indicator you must copy the Incident or Change properties and logic over to the Problem table and ensure Impacted Business Applications is being populated appropriately. As with Change and Incident, CSDM services must be configured and the form populated with at least a CI.

Portfolio TCO

Custom Script Indicator that only appears if the Total Cost of Ownership (sn_apm_tco) plugin is installed and configured. This will generate profile indicator scores when TCO values are being captured against Business Applications. The data is stored in the Total Cost of Ownership [sn_apm_tco] table.

Technical Risk

Assessment Indicator with a single Metric Category and Metric related to the technical risk of the application.

Technology Lifecycle Risk

Custom Script Indicator which requires Technology Portfolio Management (TPM) (sn_apm_tpm) plugin to be installed and the TPM app configured with Business App to App Service to Lifecycle and Technology Risk data. The data is stored in the TPM Technology Risk [sn_apm_tpm_technology_risk] table.

Total change hours

PA Indicator which calculates the total time all Change Requests referencing a given Business Application have been open for the given period. However, this is calculated via the Business Application field on the Change Request, which is deprecated as of CSDM and should no longer be used. The logic to locate Impacted Business Applications on Change Requests is already available with the Number of Changes via Service Profile Indicator, so it would make sense to edit the Summed duration of closed changes PA Indicator and add (or more usefully, remove the existing Business Application Breakdown and then add) the Number of Changes via Service Breakdown, then reference this breakdown in the Profile Indicator.

Usage

Query Condition Indicator using the Active user count field aggregated as a SUM and consolidated together by average. The challenge here is that there is no automation behind the field to populate how many users are using a Business Application, even though the documentation mentions the number of user sessions (where would you get this data from for hundreds or thousands of business applications?), so it seems rather pointless estimating or even guessing a value and then scoring on it, unless you can integrate with systems to gather this data automatically.

To me, makes more sense to use/recreate the User base Profile Indicator from the Cloud Assessment application instead, as User base is a choice field with bands of values, and is much more likely to contain reasonable data as it’s a pretty useful metric when comparing similar Business Applications.

Business Capability Scoring Profile

People

Assessment Indicator with a single Metric Category and Metric related to the people side of the capability.

Process

Assessment Indicator  with a single Metric Category and Metric related to the process side of the capability.

Technology

Assessment Indicator  with a single Metric Category and Metric related to the technology side of the capability.

Cloud Readiness Assessment Profile

Business Criticality

Custom Script Indicator that gets a score based on the “Business criticality” field on  Business Application.

Business Impact

Parent Indicator for the child Indicators Business Criticality, Core Architecture, and Strategic Importance.

Cloud Readiness

Parent Indicator for the child Indicators Cloud Version Available, Complexity of Integration, Data Classification, and Regulatory Implications.

Cloud Version Available

Custom Script Indicator that gets a score based on the “Cloud version available” field on Business Application.

Complexity of Integration

Custom Script Indicator that gets a score based on the “Complexity of Integration” field on Business Application (High, Medium, Low).

Core Architecture

Custom Script Indicator that gets a score based on the “Core architecture” tick box field on Business Application.

Data Classification

Custom Script Indicator that gets a score based on the “Data classification” field on Business Application (Confidential, Highly Sensitive, Internal, Public).

Regulatory Implications

Custom Script Indicator that gets a score based on the “Regulatory Implications” field on Business Application (Strict, Some, None).

Strategic Importance

Custom Script Indicator that gets a score based on the “Emergency tier” field on Business Application (Business Critical, Mission Critical, Non Critical).

User Base

Custom Script Indicator that scores based on the choice values in the “User base” field. Many customers change or add values to this field, so the Indicator script will need to be updated to match.

Legacy Profile Indicators

Thanks to Mark Castoe for this historic information.

If you have an older instance of ServiceNow you may have APM indicators from Financial Modeling (ITFM). These are no longer provided on new instances as the ITFM module is no longer available. These are:

• Application TCO
• Facilities Cost
• Hardware Cost
• Labor Cost
• Other Cost
• Service Cost
• Software Cost

As of March 2021, the Financial Modeling application and therefore the APM TCO and other cost indicators based upon the ITFM functionality are no longer available.

Creating Assessments

Populating the Assessment with Questions

In the baseline, the Assessment Indicators reference their own Assessment Metric Type and Metric Category, which is provided with a single dummy Metric (Question). You are expected to flesh out the assessment with desired questions – take a look at the article linked in Acknowledgements and Links for some suggested sample questions.

Assessment Distribution

Application Assessments are typically distributed to specific assessors, an assessment group, or dynamically to a referenced user/group on the target record.

If generating Assessments outside of EA, you will be prompted to create a condition and assessor field on the Assessment Metric Type, and add the same condition to each Metric Category you add, as well as generating assessable records. If you do no carry out these steps then when you click the Generate Assessments button, it will return a success message but no assessments will be generated and a message will be added to the system log (the assessment and user references on the first line with be Sys IDs):

[AssessmentCreation - createInstances_Stakeholders]: For assessment abc, user abc,  no instance is created. Things to check:
1. Please make sure that at least one metric category has at least one assessable record associated.
2. Please make sure that this category user is made as stakeholder by associating it with at least one assessable record from that metric category.
3. Please make sure that if domain plugin is installed, at least one assessable record is visible to this user.: no thrown error

Once correctly configured, this will generate an assessment for each assessable record and send it to every user referenced in the Condition.

EA Assessment Functionality

EA has its own slightly different functionality for generating targeted assessments. This is the reason why there is a Generate Assessments button on the Profile Indicator record when the Data source is Assessments. This button launches the generate_assessments UI Page, which (eventually) uses the APMAssessmentsUtil script include to generate the targeted assessment records.

Viewing Assessment Instances

You can view the generated assessments in the Profile Indicator’s Assessments Instances related list. Note that you are not able to see the user’s response in EA Workspace, but you can open the record in the legacy platform view and click the View User’s Response related link to see what the assessment will look like to the assessor.

Limitations

There is one limitation with the EA assessment functionality – you can only send an assessment with one Metric Category at a time.

If you want to get all of the assessable values that contribute to the Application Score for a Business Application, and compare and contrast them in the Bubble Chart, you must send four separate assessments out.

Whilst this is fine if you are sending each assessment out to different stakeholders, it would be useful to have a way to wrap multiple metric categories into one Assessment Type but still have them create targeted assessments.

There is a workaround, but it is time consuming, requires manual work, and is not well suited to generating periodic assessments:

1. Add the Four Metric Categories for Business Value, CSAT, Functional Fit and Performance to the Question Bank.
2. Generate one new Assessment Metric Type only.
3. Add the four Metric Categories from the Question Bank (so you are not creating new Metric Categories just for this assessment)
4. Set the Condition, Filter and User (e.g. IT Application Owner = Abel Tuter for the Condition).
5. Generate the Assessable Records – you should see only the Business Applications that match the Condition.
6. Then add the same Condition in the Metric Type to each Metric Category – after saving you should see the related list of Assessable Records populate.
7. Click Generate Assessments on the Metric Type.
8. Now repeat steps 4-7 for each subsequent stakeholder (in this case, IT Application Owner), modifying the Condition on the Metric Type and related Metric Categories, and generating more assessments.
9. Finally, amend the Profile Indicator for each category and replace the existing Assessment Metric type with the one created in Step 2 above, choosing the same Metric Category that was previously populated. Do not click Generate Assessments on the Profile Indicator record.

The only other option (generating multiple Metric Types, one for each stakeholder) is not workable as each Profile Indicator must reference a single Metric Type and Metric Category, and you want one single Profile Indicator for each Metric Category in order to have meaningful scoring data in the Bubble Chart.

I understand that Smart Assessments will be coming to EA, but for now this is the only meaningful option to both have one assessment, and ensure that the Bubble Chart in the Application Rationalization section of EA Workspace is populated with application scores for each Metric Category.

Application Scoring

Once you have all the Profile Indicators set up and related to your Application Scoring Profile, it is time to set a Weightage value for each indicator. This is then fed into the scoring engine and a weighted and normalized score is calculated for each target record (Business Application or Business Capability).

The application scoring engine calculates a Normalized Value for each Indicator based on a relative ranking of 1 to 10, where a higher value denotes a better score. This is calculated as follows:

(((appWeight-minWeight) / (maxWeight-minWeight)) x9) +1)

Where:

• Application Weight (appWeight) is the relative percentage weightage of an indicator as part of the overall application score
• Minimum Weight (minWeight) is the target minimum weightage required for the indicator
• Maximum Weight (maxWeight) is the target maximum weightage

If target minimum or target maximum are not defined against an indicator, then the target maximum will be the highest score obtained, and the target minimum will be the lowest score obtained across the data set.

The Application Score is calculated as a sum of all the Application Indicator Normalized Values.

The table below contains an example. The % sign is in brackets below because you can assign summed Weightages above a total of 100 and the calculation will still work.

Indicator	Weightage	Normalized Score	Application Score
TCO	40(%)	9.6	9.6 x 0.4 = 3.84
Process	25(%)	10	10 x 0.25 = 2.50
People	25(%)	7	7 x 0.25 = 1.75
Facilities Cost	10(%)	8	8 x 0.1 = 0.8
Total			8.89

Scoring Configuration

Note that in order to generate scores, you need to configure and run the following:

• PA Premium needs to be installed and collection job [PA Incident] Daily Data Collection running as it uses the Number of Changes via Service, Number of Incidents via Service, and Number of Problems via Service Breakdowns from the related Number of new incidents Indicator.
• You may also need to run the [PA Incident] Historic Data Collection job as well.
• Run scheduled job Load Application Indicators and compute Application Scores
• PA collection jobs (to populate trend-based indicator data on the Application 360 - EA Workspace Dashboard):
  • PA App 360 Daily Data Collection
  • PA App 360 Historic Data Collection

Application Scores

Once generated, application scores are stored in the Indicator Score [apm_app_indicator_score] table. These values are visible from the List button in the Application Rationalization section of EA Workspace.

Regenerating Scores

The Profile Indicator form has a Regenerate Scores button that enables you to generate scores again for a given fiscal period.

Assessment Scores

Scores will not be calculated for assessment-based indicators until all the assessments in the assessment GROUP have been either completed or cancelled.

Scores will also only be calculated for assessments that were CREATED within the fiscal period you selected when generating the scores. It doesn't matter when the assessment instances were completed, it matters when they were created.

EA Workspace Visualisations

Application Assessment

The data is visible in these EA Workspace locations:

• Home
  • Application Portfolio > Insights section - note that the profile indicators are set up for quarterly assessments, but if you do not assess quarterly, these Insights might not populate. In order to sync your process with the Insights data, update System Property com.glide.fiscal_calendar.fiscal_unit. This is set to “quarter” by default and the possible options are listed in the Choices field.
  • Portfolio Health section: Business applications not assessed card and Assessments tab under Needs attention.
• Dashboard – Pick the Application 360 – EA Workspace Dashboard.
• Application Rationalization – Bubble Chart and List views. List is where you can view the Overall Score (application score) plus all the (current) individual Profile Indicator Scores for a Business Application.

Business Capability Assessment

The data is visible in these EA Workspace locations:

• Home – Portfolio Health section: Business capabilities not assessed card and Assessments tab under Needs attention.
• Dashboard – Business Portfolio tab: Business Capabilities with Low Scores report

Cloud Readiness Assessment

The data is visible in these EA Workspace locations:

• Home – Assessments tab under Needs attention.
• Application Rationalization – Bubble Chart (pick the Cloud Readiness option)

Configuring and Using the Bubble Chart

The Application Rationalization section of EA Workspace contains a Bubble Chart, which allows you to compare and contrast values for business applications in an Invest/Sustain/Migrate/Retire matrix. This list appears fixed at first; however, a table not accessible from the UI – Application Bubble Chart [apm_bubble_chart] – allows you to choose any selection of two or even three Profile Indicators for comparison. Note that there is a limitation with EA Workspace – although you can associate indicators with any Application Profile, only those that are related to the Default Application Profile will be accessible.

Thanks to Mark Castoe:

In order to specify another Scoring profile, you will need to create (or update) the system property, sn_apm_ws.app_indicator_scoring_profile, and set the sys_id of your new/custom scoring profile as the value of the system property.  This will tell the bubble chart mechanism to use your scoring profile instead of the default.

Summary

Application Assessment is a powerful tool that contributes to accurately scoring Business Applications, but it comes with some dependencies, such as CSDM implementation, and some limitations with assessments. 

Key Takeaways

• Understand the architecture of Application Scoring and Assessment.
• Be aware of the limitations of existing Profile Indicators.
• Understand the limitation of single-category assessments.
• Be aware of the requirement to flesh out the assessments with questions relevant to your use cases.
• Understand how applications are scored.
• Know where to locate data visualisations related to assessment and scoring.

Acknowledgements and Links

Thanks to @mcastoe for his excellent article Measure your Application Inventory with the Indicator and Scoring Framework as this contains a lot of valuable historical, current and potential future information, including some sample questions to include with the provided Assessment Categories.

Thanks also to @MikeWilsonNYC  for his suggestion on generating assessments with only one metric type, and valuable information about assessment-derived scores.

ServiceNow EA Application Assessment product documentation: https://www.servicenow.com/docs/bundle/zurich-application-portfolio-management/page/product/applicat...