Now Assist Guardian analytics
Summarize
Summary of Now Assist Guardian Analytics
The Now Assist Guardian analytics dashboard is designed for administrators to monitor the performance of guardrails that protect against offensive content and prompt injection when interacting with large language models (LLM). It provides insights into guardrail effectiveness through various performance indicators.
Show less
Key Features
- Average Latency: Measures latency due to active guardrails, helping to identify periods of high guardrail activity.
- Offensive Content Indicators: Includes metrics such as the percentage flagged as offensive, total occurrences, and breakdowns by category and skill.
- Prompt Injection Indicators: Similar metrics to offensive content, including latency, percentage flagged, total occurrences, and occurrences tracked by skill.
- Dashboard Filters: Allows viewing guardrail activity across specific skills and date ranges for tailored analysis.
Key Outcomes
By utilizing the Now Assist Guardian analytics dashboard, customers can effectively track and analyze guardrail performance, enabling them to enhance the safety and reliability of LLM interactions. This monitoring aids in identifying trends, making informed adjustments, and ensuring compliance with content standards.
Monitor the performance of guardrails enabled through Now Assist Guardian.
The Now Assist Guardian analytics dashboard helps admins monitor and evaluate the effectiveness of offensive content and prompt injection guardrails in tracking and analyzing requests sent to large language models (LLM) and their responses.
- Average latency as a result of active offensive content and prompt injection guardrails. High latency could mean increased guardrail activity in the period.
- Count and percentage of offensive content and prompt injection occurrences.
- Skills where offensive content and prompt injection occurrences were detected.
Apply the filters on the dashboard to view guardrail activity for skills in a date range. See Now Assist Analytics dashboard indicator details for information on the data and calculations behind each indicator.
Offensive content indicators
- Guardrail-added latency
- This area of the dashboard shows the average latency as a result of the active offensive content guardrail for the selected skills and date range.
Figure 2. Guardrail-added latency indicator - Percentage flagged as offensive
- This area of the dashboard shows the percentage of requests and responses to and from the LLM service that are flagged for offensive content.
Figure 3. Percentage flagged as offensive indicator - Total offensive content occurrences
- This area of the dashboard shows the total number of offensive content occurrences for the selected skills and date range.
Figure 4. Total offensive content occurrences indicator - Categories of offensive content
- This area of the dashboard shows a breakdown of offensive content occurrences by the categories. If content is deemed to be offensive under more than one category, for example, toxic and defamatory, the occurrence is counted
individually toward both the categories. For more information on offensive content categories, see Now Assist Guardian.
Figure 5. Categories of offensive content indicator - Offensive content occurrences by skill
- This area of the dashboard shows the number of offensive content occurrences over time by the skills in which the content is detected.
Figure 6. Offensive content occurrences by skill indicator
Prompt injection indicators
- Guardrail-added latency
- This area of the dashboard shows the average latency as a result of the active prompt injection guardrail for the selected skills and date range.
Figure 7. Guardrail-added latency indicator - Percentage flagged as prompt injection
- This area of the dashboard shows the percentage of requests and responses to and from the LLM service that are flagged for offensive content.
Figure 8. Percentage flagged as prompt injection indicator - Total prompt injection occurrences
- This area of the dashboard shows the total number of offensive content occurrences for the selected skills and date range.
Figure 9. Total prompt injection occurrences indicator - Prompt injection occurrences by skill
- This area of the dashboard shows the number of prompt injection occurrences over time by the skills where prompt injection attempts were detected.
Figure 10. Prompt injection occurrences by skill indicator