---
sourceDocument: Australia Enable AI
sourceDocumentLink: https://www.servicenow.com/docs/r/australia/intelligent-experiences

 Release :

    - australia

ft:locale :

    - en-US

ft:publication_title :

    - Australia Enable AI

ft:clusterId :

    - platai

bundleId :

    - platai

workflow :

    - Platform


---

# Prompt evaluation guidelines

# Prompt evaluation guidelines {#ariaid-title1}

Release version: Australia  
Updated September 10, 2026  
![](https://www.servicenow.com/docs/portal-asset/ico-clock) 2 minutes to read
Summarize  
![AI sparkle icon](https://servicenow.com/docs/portal-asset/ai-sparkle-icon) Summarized using AI  
This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.  

## Summary of Prompt evaluation guidelines

These guidelines help ServiceNow customers effectively evaluate AI skill prompts during development and prior to deployment.
Proper evaluation ensures that prompts perform well on relevant data, avoid overfitting, and meet quality standards suited to your use case.
Show full answer Show less  

## Evaluation approach

* **Ongoing evaluation during development:** Continuously assess prompt outputs on batches of test data to iteratively improve the prompt. Avoid relying on only one or two examples to prevent misleading conclusions from noise.
* **Final performance evaluation before deployment:** Test the prompt on a fully isolated and representative batch of "test" data that was not used during development. This prevents prompt overfitting, where the prompt becomes too specialized to the development examples, inflating performance estimates.
* **External evaluation:** Copy model-generated responses for assessment outside the AI Skill Kit to maintain objectivity and flexibility in analysis.

## Evaluation metrics

Select metrics based on the prompt's use case and output type:

* **Classification-based assessment for short outputs:** When outputs are concise and have well-defined correct answers (e.g., true/false, multiple choice), use metrics like precision, recall, and F1 scores for quantitative evaluation.
* **Human scoring for longer generative outputs:** For longer texts where multiple correct answers exist, human evaluators should rate outputs on scales (e.g., 1 to 5) across key dimensions:
  * **Faithfulness:** Does the output accurately reflect the provided context without hallucinating irrelevant information?
  * **Correctness:** Is the output factually and instructionally accurate?
  * **Helpfulness:** Does the output effectively assist with the intended task, considering end-user needs?
* **Fluency:** Is the text grammatically correct, coherent, and free of typographical errors?  
Evaluate a skill prompt during development and again before deployment, using batches of test data and metrics that suit your use case.

## Evaluation approach {#evaluating-the-prompt__section_dss_msh_ccc}

To determine the effectiveness of your prompt, you should evaluate batches of test data. You should copy the model-generated responses and perform evaluations outside of AI Skill Kit.

## During prompt development {#evaluating-the-prompt__section_twb_psh_ccc}

Ongoing, improvised evaluation should take place alongside the development of the prompt. This ongoing evaluation enables you to adapt the prompt based on observed model outputs. Testing a change to a prompt against one or two
examples can be tempting. To avoid reacting to noise, look at larger batches, and consider the statistical significance of the performance differences that you observed.

## Final performance evaluation {#evaluating-the-prompt__section_y5h_zsj_ccc}

Before you deploy a skill, you should test the prompt on a representative batch of data that was isolated from the development process, that is, "test" data. You want to use isolated test data because of a phenomenon known as
prompt overfitting. Iteratively editing a prompt based on the model outputs generated on the same data that is used for testing can lead to significant overestimates of performance. This result is because the prompt can become
overspecialized to the specific examples used in development. Even though the effect is typically less dramatic than what occurs when fitting machine learning model parameters to a test dataset, it's rooted in the same underlying
principles, and should be avoided.

## Evaluation metrics {#evaluating-the-prompt__section_khf_1tj_ccc}

Selecting the right metrics for evaluation is an important consideration. The following list provides a few approaches, each of which may be more or less appropriate depending on the use case.  
* Classification-based assessment of short generationsThis approach requires labeled records, and it works best when the labels are short, well-defined "right answers," for example, true or false, multiple-choice, or category
  selection. In these cases, the model outputs can usually be parsed and formatted, then metrics like precision, recall, F1 scores, and so on can be directly calculated.

* Assessment of longer generationsMany generative AI use cases require longer model generations, and there are many possible "right answers." In these cases, the output can be scored (by human evaluators) along several
  different axes, for example:

  * FaithfulnessIs the generated text faithful to the context provided in the skill prompt? (The opposite of faithfulness is hallucination, which is to say that the model injects out-of-context information.)

  * CorrectnessIs the generated text correct relative to the skill instruction?

  * HelpfulnessIs the generated text helpful relative to the task that the skill performs? (Helpfulness is subjective but it's important to try to measure. Doing so properly requires a solid understanding of the needs of
    the people who ultimately use the skill.)

  * FluencyIs the generated text grammatically correct? Does it have any typos, issues with coherency, and so on?

  {#evaluating-the-prompt__ul_yx1_c5j_ccc}  
  Note:  
  It's useful to score these properties on a scale, such as 1 through 5, rather than with yes or no.
{#evaluating-the-prompt__ul_dsl_xtj_ccc}