--- sourceDocument: Australia IT Service Management sourceDocumentLink: https://www.servicenow.com/docs/r/australia/it-service-management Release : - australia ft:locale : - en-US ft:publication_title : - Australia IT Service Management ft:clusterId : - itsm bundleId : - itsm workflow : - Technology --- # Quality assessment skill # AI Quality assessment for L1 IT Service Desk AI Specialist {#ariaid-title1} Release version: Australia Updated September 18, 2026 ![](https://www.servicenow.com/docs/portal-asset/ico-clock) 2 minutes to read Summarize ![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete. ## Summary of AI Quality assessment for L1 IT Service Desk AI Specialist The AI Quality Assessment skill is designed to evaluate coaching assessments for L1 IT Service Desk AI Specialists by scoring responses against a predefined rubric. This skill is part of the L1 IT Service Desk AI Specialist plugin and specifically assesses how well AI-generated responses address IT service incidents. Show full answer Show less ## What the Skill Evaluates The skill analyzes several inputs, including incident context (number, short description, description) and the AI Specialist's response (comments, work notes, close notes, and references to knowledge articles or catalog items). It scores responses based on four metrics: * **Resolution Step Quality** (contributes to overall quality): Measures if each step helps resolve the incident. * **Executable Resolution** (contributes to overall quality): Assesses whether following the response would resolve the incident. * **Knowledge Accuracy** (diagnostic only): Checks consistency between cited knowledge articles and the response. * **Root Cause of Failure** (diagnostic only): Identifies the main reason if the resolution fails. The diagnostic metrics provide insight but do not affect the overall quality score. ## Scoring and Quality Tiers The overall quality score is calculated by averaging the Resolution Step Quality and Executable Resolution metrics, with the score limited to a minimum of zero. Executable Resolution is scored as: * Yes = 100 * Partial = 50 * No = 0 Based on the composite score, the response is categorized into quality tiers: * **Excellent:** 85--100 * **Good:** 70--84 * **Partial:** 50--69 * **Poor:** 0--49 ## Roles and Configuration Users assigned the `sncoaching.coach` or `snsowitsmcommon.snservicedeskmanager` roles can view quality assessment results. The underlying AI prompts are editable through the NowAssist Skill Kit. The skill uses a large language model (LLM) to generate scores and supports integration with third-party AI models as well as the native ServiceNow AI Platform LLM. The AI Quality Assessment skill evaluates the work captured in a coaching assessment against the rubric defined on the coaching opportunity. The IT Service Management AI Quality Assessment skill evaluates the work captured in a coaching assessment and scores it against the rubric defined on the associated coaching opportunity. The skill is part of the L1 IT Service Desk AI Specialist plugin. ## What the skill evaluates {#qa-l1-sd-ai-spec__section_qa_inputs} The skill evaluates the following information as input: * Incident context, such as the number, short description, and description. * The L1 IT Service Desk AI Specialist response, including comments, work notes, close notes, and any referenced knowledge articles or catalog items. {#qa-l1-sd-ai-spec__ul_qa_inputs} The skill scores the response against four metrics. Two metrics contribute to the overall quality score, and two are diagnostic only: * **Resolution Step Quality**: whether each step contributes to resolving the incident. Contributes to the quality score. * **Executable Resolution**: whether the response would resolve the incident if followed. Contributes to the quality score. * **Knowledge Accuracy (Diagnostic)**: whether cited knowledge articles are consistent with the response. Doesn't affect the quality score. * **Root Cause of Failure (Diagnostic)**: the dominant reason a resolution fell short. Doesn't affect the quality score. {#qa-l1-sd-ai-spec__ul_qa_categories} The assessment related list shows a human-readable Value column with each metric's label. The score column is blank for diagnostic metrics. On the coaching assessment form, the Quality metric results related list shows the Assessment, Metric, Score, Value, and Reasoning columns. The Category and Metric type columns no longer appear. The Score column is empty for diagnostic metrics. The Value column shows a human-readable label for each metric, as listed in the following table. {#qa-l1-sd-ai-spec__table_qa_value_labels__entry__2} | Metric | Value labels | |-|-| | Resolution Step Quality | Contributes to resolution, Neutral, Misleading, Not Accessible | | Executable Resolution | Yes, Partial, No | | Knowledge Accuracy (Diagnostic) | Consistent, Partially Consistent, Contradicts KB, No KBs Cited | | Root Cause of Failure (Diagnostic) | None, Execution Blocked, Knowledge Gap, Wrong Diagnosis | [Table 1. Value labels by metric] {#qa-l1-sd-ai-spec__table_qa_value_labels} ## Quality score and tier {#qa-l1-sd-ai-spec__section_qa_scoring} The skill averages Resolution Step Quality and Executable Resolution to calculate the composite quality score, clamped at 0, and assigns a quality tier based on that score. Diagnostic metrics don't affect the composite score. Executable Resolution is scored from the AI judge's final verdict: Yes (100), Partial (50), or No (0). {#qa-l1-sd-ai-spec__table_qa_tiers__entry__2} | Score range | Tier | |-|-| | 85--100 | Excellent | | 70--84 | Good | | 50--69 | Partial | | 0--49 | Poor | [Table 2. Quality tier thresholds] {#qa-l1-sd-ai-spec__table_qa_tiers} ## Roles and configuration {#qa-l1-sd-ai-spec__section_qa_access} Users with the sn_coaching.coach or sn_sow_itsm_common.sn_service_desk_manager role can access quality assessment results. The underlying prompts can be edited in the NowAssist Skill Kit. The skill calls an LLM (large language model) to generate the score and supports third-party (3P) models in addition to the ServiceNow AI Platform native LLM.