Access and Reporting on Generative AI Logs (sys_generative_ai_log)

[mouradhadj-]

Hello Community,

I am working on developing Custom Reporting and Performance Analytics (PA) KPIs for our Now Assist adoption and usage. I have identified two key tables for reporting:

1. sys_gen_ai_usage_log (The main usage log for billing and high-level activity).

2. sys_generative_ai_log (A deeper log table that seems to contain more granular data).

I've encountered a security issue with the second table and have some related questions on its reporting utility and risks.

---

1. Access Issue with sys_generative_ai_log

When attempting to view or report on the sys_generative_ai_log table, I receive a security constraint message (as shown below).

"Security restricted access to data in the table Generative AI Log [sys_generative_ai_log]. Number of rows removed from this list by Security constraints: 20"

My Question:

• What specific ACL/Role is typically required to gain read access to the sys_generative_ai_log table? I have already added the main usage table (sys_gen_ai_usage_log) to the glide.ui.permitted_tables property, but this deeper log seems to have additional restrictions.

---

2. Reporting Utility: Counting Skills per Ticket

Assuming I gain access, I want to confirm if the sys_generative_ai_log table can solve a specific reporting challenge.

The main usage table (sys_gen_ai_usage_log) is great for counting unique tickets where AI was used (COUNT DISTINCT on the Document field). However, I want to know the total number of individual skills or use cases triggered per ticket (e.g., Ticket INC123 had 'Summarize' (1) and 'Suggest Resolution' (1) = 2 Skills/Use Cases used).

My Question:

• Does the sys_generative_ai_log table contain a more granular record of each skill trigger or AI step that would allow me to accurately calculate the "Average number of use cases used per ticket"?

---

3. KPI Ideas and Data Risks

I'm looking for suggestions on valuable KPIs using this granular data. I am already implementing:

• Adoption: % of tickets resolved with at least one use case used.

My Questions:

• KPI Suggestions: Does anyone have suggestions for "Value" or "Quality" KPIs using the detailed data from sys_generative_ai_log? (e.g., correlating logs with the Feedback field).

• Data Access Risks: Given the sensitivity of the data (prompts and potential responses are logged), are there any security or performance risks to be aware of when granting broad read or report_on access to the sys_generative_ai_log table?

Thank you in advance for any insights!

Best regards,

[MaxMixali]

Access and Reporting on Generative AI Logs (sys_generative_ai_log) – ServiceNow

1. Understanding the Tables
--------------------------------------------------
ServiceNow tracks AI interactions in two related tables:

- sys_gen_ai_usage_log: Primary table for tracking Now Assist usage, including billing metrics, request type, and document linkage.
- sys_generative_ai_log: Detailed log of individual AI activities — each record represents a skill invocation, prompt, or response event for a specific case, incident, or task.

2. Access Restrictions and ACLs
--------------------------------------------------
The sys_generative_ai_log table is security-restricted by default because it may store sensitive prompt and response text (potentially including user or customer data).

Even security_admin role does not automatically grant access.

To allow access:
1. Navigate to System Security → Access Control (ACL)
Search for: Table = sys_generative_ai_log
2. Identify the ACLs:
- sys_generative_ai_log.* (field-level)
- sys_generative_ai_log.read (table-level)
3. Clone the ACL and add a reporting role (e.g., sn_gen_ai.reporter or custom x_company_ai_analytics).

Alternative:
Grant temporary read access using a UI Policy Override or Read Replica Scripted REST API, allowing specific roles to query anonymized data.

3. Reporting Utility and Use Case
--------------------------------------------------
Yes — the sys_generative_ai_log table provides granular event-level records that can support advanced KPIs:

Key Fields:
- document: Record (Incident, Case, Task) associated with the AI interaction
- skill: Name of the AI skill used (e.g., Summarize, Suggest Resolution)
- user: User who triggered the interaction
- execution_time: Timestamp for skill usage
- result: Status or response metadata

Example Reporting:
- Average number of skills per record: COUNT(skill) / COUNT(DISTINCT document)
- Most-used skills per team: GROUP BY assignment_group
- Custom skill adoption: Filter on custom skill names

4. KPI Suggestions
--------------------------------------------------
| KPI Name | Formula | Description |
|-----------|----------|--------------|
| AI Adoption Rate | % of tickets where ≥1 skill used | Measures Now Assist usage across operations |
| Average Use Cases per Ticket | Total Skills ÷ Distinct Tickets | Indicates depth of AI interaction |
| Custom Skill Utilization | Count(custom_skill) | Tracks use of enterprise-defined AI capabilities |
| AI-Driven Resolution Rate | Tickets resolved with Now Assist ÷ Total tickets | Assesses automation efficiency |
| Feedback Score Trend | Average(feedback_score) per skill | Monitors user satisfaction and AI accuracy |

5. Security and Data Risks
--------------------------------------------------
Be cautious when broadening access:
- Data Sensitivity: Prompts and responses might include user-entered or system data.
- Performance Impact: The table can grow rapidly due to high transaction volume.

Best Practices:
- Restrict access to analytics roles only.
- Create ETL scripts that extract sanitized data into a reporting table (excluding prompts/responses).
- Avoid joining large AI tables with core transactional tables in real time.

6. Recommended Next Steps
--------------------------------------------------
1. Request the sn_gen_ai.admin or sn_gen_ai.reporter role from your ServiceNow administrator.
2. Confirm if your instance’s data governance policy allows access to AI logs.
3. Build PA indicators on the sys_gen_ai_usage_log table for high-level KPIs.
4. For granular insights (skills, usage by group), replicate the sys_generative_ai_log data into a reporting-safe table.

7. Summary
--------------------------------------------------
Key Takeaways:
- sys_generative_ai_log is restricted — use ACL cloning or custom roles for access.
- It contains detailed skill-level AI events, enabling advanced KPI reporting.
- Always implement data governance and anonymization before granting reporting access.
- Recommended: Use ETL extraction + Performance Analytics for scalable dashboards.

Best Practice:
For regulated industries (finance, HR, healthcare), build a “sanitized AI Analytics” table to avoid exposing sensitive prompts or personal data during reporting.

[mouradhadj-]

Hello @MaxMixali ,

Thanks so much for those details. i will try your solution.
Best regard

[MaxMixali]

ServiceNow – Access & Reporting on Now Assist Logs (sys_generative_ai_log) with KPI Ideas & Data Risks

Tables in scope
- sys_gen_ai_usage_log: High-level usage/billing; good for adoption counts and distinct-ticket metrics.
- sys_generative_ai_log: Granular, event/skill-level telemetry (prompts/steps/outcomes). Security-restricted by design.

1) Access to sys_generative_ai_log (ACLs & constraints)
------------------------------------------------------
Why you see “Security restricted … rows removed by Security constraints”:
- Table has strict ACLs and often row-level security constraints because entries may include prompts/responses and sensitive context.
- glide.ui.permitted_tables only affects UI list permission prompts; it does NOT bypass ACLs or row constraints.

What typically grants read:
- Instance-specific ACLs (table.* and read) often require a role like sn_gen_ai.admin/reader or a custom analytics role. There is no universal role name—review ACLs in your instance.
Steps:
1. System Security → Access Control (ACL) → filter Name starts with sys_generative_ai_log.
- Review table.* and table.read ACLs (script + required roles).
2. If “Security constraints” are configured, check Contextual Security / Security Constraints for this table and add a targeted allow for your analytics role/group.
3. Safer pattern: create a nightly ETL that copies minimal, anonymized columns into a custom table (e.g., u_gen_ai_event_fact) and grant report_on/read there.

Tip: Even with security_admin, row constraints can still hide rows. Don’t rely on permitted_tables for this use case.

[MaxMixali]

ServiceNow – Access & Reporting on Now Assist Logs (sys_generative_ai_log) with KPI Ideas & Data Risks

Tables in scope
- sys_gen_ai_usage_log: High-level usage/billing; good for adoption counts and distinct-ticket metrics.
- sys_generative_ai_log: Granular, event/skill-level telemetry (prompts/steps/outcomes). Security-restricted by design.

1) Access to sys_generative_ai_log (ACLs & constraints)
------------------------------------------------------
Why you see “Security restricted … rows removed by Security constraints”:
- Table has strict ACLs and often row-level security constraints because entries may include prompts/responses and sensitive context.
- glide.ui.permitted_tables only affects UI list permission prompts; it does NOT bypass ACLs or row constraints.

What typically grants read:
- Instance-specific ACLs (table.* and read) often require a role like sn_gen_ai.admin/reader or a custom analytics role. There is no universal role name—review ACLs in your instance.
Steps:
1. System Security → Access Control (ACL) → filter Name starts with sys_generative_ai_log.
- Review table.* and table.read ACLs (script + required roles).
2. If “Security constraints” are configured, check Contextual Security / Security Constraints for this table and add a targeted allow for your analytics role/group.
3. Safer pattern: create a nightly ETL that copies minimal, anonymized columns into a custom table (e.g., u_gen_ai_event_fact) and grant report_on/read there.

Tip: Even with security_admin, row constraints can still hide rows. Don’t rely on permitted_tables for this use case.

2) Can it support “skills per ticket”? (Yes)
--------------------------------------------
- sys_generative_ai_log commonly includes: table, document/document_sys_id, capability/skill, user, created_on, status/outcome, channel, (optionally) prompt/response metadata.
- Join by (table, document/document_sys_id) to your ticket (e.g., incident, sc_req_item).
- Compute per-ticket distinct skills:
- For each ticket: COUNT DISTINCT(skill/capability)
- Average across tickets: AVG(distinct_skills_per_ticket)
- If direct access is restricted, push the following to a safe fact table:
- ticket_table, ticket_sys_id, skill, created_on, user, outcome
- Exclude prompt/response text

3) KPI suggestions (Adoption, Value, Quality, Operations)
---------------------------------------------------------
Adoption
- % tickets with ≥1 AI skill used (by group, priority, channel)
- % active agents using AI weekly (DAU/WAU)
- Skill mix distribution (summarize, suggest_resolution, draft_comment, …)

Value / Outcome
- Avg # of use cases per AI-enabled ticket
- MTTR improvement comparison between AI-enabled and standard tickets
- % AI-enabled tickets resolved within SLA compared to baseline
- Time-saved proxy (by skill) using standard benchmarks
- Deflection rate (cases resolved without escalation after AI step)

Quality / Experience
- Suggestion acceptance rate (accepted / offered)
- Feedback score trend (thumbs up/down or 1–5) per skill and per group
- Reopen rate for AI-enabled tickets vs baseline
- Error/timeout rate per skill/channel

Operations
- Top skills by effectiveness (acceptance × SLA success)
- Outlier analysis: high error rate skills, groups, or channels
- Usage heatmap by hour/day; capacity planning

PA implementation tips
- Indicator sources: usage table for adoption; granular log (or safe fact) for skill-level metrics.
- Breakdowns: Assignment group, Agent, Capability/skill, Channel, Priority, Category/Service.
- Targets/Thresholds: phased adoption targets and quality gates by release wave.
- Avoid heavy joins on raw logs in live widgets; pre-aggregate nightly for PA.

4) Data access & performance risks (and mitigations)
----------------------------------------------------
Risks
- Sensitive content (prompts/responses) exposure.
- Large volume; ad-hoc queries can harm performance.
- Cross-domain data visibility (multi-department environments).

Mitigations
- Create a curated, anonymized fact table (no prompt/response text; keep IDs and metrics). Grant read/report_on only to analytics roles.
- Add indexes: (table, document, created_on), (skill), (user), as needed.
- Retention/archival: keep raw logs short (e.g., 30–90 days), retain summarized facts longer.
- Field-level security: if you must keep text, protect with field ACLs or Vault/column encryption.
- Governance: document lineage and access; review quarterly.

5) Suggested operating model
----------------------------
- Short term: request appropriate role (per ACL review) for a limited analytics group; prototype KPIs.
- Medium term: implement nightly ETL to u_gen_ai_event_fact with safe columns and indexes; build PA from that.
- Long term: maintain benchmarks for “time-saved” per skill; evolve targets and quality thresholds by release wave.

TL;DR
-----
- Access: sys_generative_ai_log is restricted via ACLs and sometimes row constraints—review ACLs, or build a sanitized analytics fact table.
- Yes, it supports “skills per ticket.” Use distinct capability counts per document.
- KPIs: Adoption, Value, Quality, Ops. Protect sensitive data, pre-aggregate, and index for PA.

[mouradhadj-]

Hello @MaxMixali ,

i tried your solution by cloning the ACL but i still don't have read  access to the table.

My question can i create a reports, KPIs Or indicator etc ... for example without having read access to the table ? 
thanks in advance