Now Assist Guardian analytics
Summarize
Summary of Now Assist Guardian analytics
Now Assist Guardian analytics provides ServiceNow administrators with a dashboard to monitor and evaluate the effectiveness of guardrails against offensive content and prompt injection in interactions with large language models (LLMs). This enables admins to track requests sent to LLMs and their responses, helping to ensure safer and more reliable AI-assisted interactions.
Show less
Key Features
- Latency Tracking: Measures average latency added by active offensive content and prompt injection guardrails, where higher latency may indicate increased guardrail activity.
- Offensive Content Metrics: Displays the count and percentage of requests flagged for offensive content, with a detailed breakdown by offensive content categories (such as toxic or defamatory), including occurrences by skill over time.
- Prompt Injection Metrics: Shows the count and percentage of requests flagged for prompt injection attempts, as well as occurrences by skill over time.
- Filter Capability: Allows filtering guardrail activity data by skills and date ranges to analyze specific segments.
Practical Use for ServiceNow Customers
By leveraging the Now Assist Guardian analytics dashboard, customers can:
- Gain insights into how guardrails impact LLM performance and latency.
- Identify and quantify offensive content and prompt injection attempts to improve AI content safety.
- Monitor which specific skills are affected by offensive content or prompt injection, enabling targeted action.
- Use filtering to analyze trends over time and across different skill sets, supporting informed decision-making on guardrail configurations and improvements.
This analytics functionality is essential for maintaining the integrity and trustworthiness of AI-powered services within ServiceNow environments.
Monitor the performance of guardrails enabled through Now Assist Guardian.
The Now Assist Guardian analytics dashboard helps admins monitor and evaluate the effectiveness of offensive content and prompt injection guardrails in tracking and analyzing requests sent to large language models (LLM) and their responses.
- Average latency as a result of active offensive content and prompt injection guardrails. High latency could mean increased guardrail activity in the period.
- Count and percentage of offensive content and prompt injection occurrences.
- Skills where offensive content and prompt injection occurrences were detected.
Apply the filters on the dashboard to view guardrail activity for skills in a date range. See Now Assist Analytics dashboard indicator details for information on the data and calculations behind each indicator.
Offensive content indicators
- Guardrail-added latency
- This area of the dashboard shows the average latency as a result of the active offensive content guardrail for the selected skills and date range.
Figure 2. Guardrail-added latency indicator - Percentage flagged as offensive
- This area of the dashboard shows the percentage of requests and responses to and from the LLM service that are flagged for offensive content.
Figure 3. Percentage flagged as offensive indicator - Total offensive content occurrences
- This area of the dashboard shows the total number of offensive content occurrences for the selected skills and date range.
Figure 4. Total offensive content occurrences indicator - Categories of offensive content
- This area of the dashboard shows a breakdown of offensive content occurrences by the categories. If content is deemed to be offensive under more than one category, for example, toxic and defamatory, the occurrence is counted
individually toward both the categories. For more information on offensive content categories, see Now Assist Guardian.
Figure 5. Categories of offensive content indicator - Offensive content occurrences by skill
- This area of the dashboard shows the number of offensive content occurrences over time by the skills in which the content is detected.
Figure 6. Offensive content occurrences by skill indicator
Prompt injection indicators
- Guardrail-added latency
- This area of the dashboard shows the average latency as a result of the active prompt injection guardrail for the selected skills and date range.
Figure 7. Guardrail-added latency indicator - Percentage flagged as prompt injection
- This area of the dashboard shows the percentage of requests and responses to and from the LLM service that are flagged for offensive content.
Figure 8. Percentage flagged as prompt injection indicator - Total prompt injection occurrences
- This area of the dashboard shows the total number of offensive content occurrences for the selected skills and date range.
Figure 9. Total prompt injection occurrences indicator - Prompt injection occurrences by skill
- This area of the dashboard shows the number of prompt injection occurrences over time by the skills where prompt injection attempts were detected.
Figure 10. Prompt injection occurrences by skill indicator