Improving ServiceNow Problem Management and Control-M Incident Automation — Seeking Best Practices

[swathik5566]

Seeking Recommendations: Automating Problem Management, Incident Tracking, and L1 Operations in ServiceNow and Control-M

Hello everyone,

I’m looking for recommendations and feasible solutions to improve incident and problem management processes in an L1 production support environment using ServiceNow (SNOW) and Control-M.

We have identified the following challenges and would appreciate insights into existing features, automation possibilities, best practices, or recommended approaches.

1. Inconsistent PRB Creation and Incident-to-Problem Association

Problem statement:

L1 operators monitor multiple dashboards to identify anomalies and create incidents for alerts that require investigation. However, recurring issues are not always identified consistently, resulting in incidents being closed without being associated with an existing Problem Record (PRB). In some cases, a PRB is not created at all, even when the same issue occurs repeatedly.

Impact:

- Recurring issues remain unresolved for extended periods because they are not formally tracked through problem management.
- Development teams may not have sufficient visibility into recurring issues to prioritize root cause analysis and permanent fixes.
- Repeated incidents may continue to be handled individually without addressing the underlying cause.

Questions and requirements:

1. How can we ensure that L1 operators consistently associate incidents with the appropriate existing PRBs?
2. Can PRBs be automatically created based on incident descriptions, error messages, recurring patterns, or incident frequency? If so, what approaches or ServiceNow capabilities would be suitable?
3. Can we implement automated rules to identify recurring incidents and recommend an existing PRB or create a candidate PRB for review?
4. What controls, workflows, or validation mechanisms can prevent incidents from being closed without the appropriate PRB association when required?
5. How can we generate reports from ServiceNow showing the number of PRBs assigned to specific teams, their age, priority, recurrence frequency, and closure status?
6. How can we ensure that PRBs are prioritized appropriately and tracked through root cause analysis, permanent fixes, and closure?

2. Incident Attribution, L1 Team Ownership, and Effort Tracking

Problem statement:

Incidents are created through multiple portals, dashboards, and server pages. Since many of these interfaces do not capture the actual user's login information, the incident's *Requested By* or *Caller* field is often populated with a generic user account.

Additionally, incident ownership may change multiple times due to reassignment between teams. As a result, it becomes difficult to identify which team originally detected and created an incident, measure the number of incidents generated by each team, and understand which applications, domains, or processes experience the most issues.

Impact:

- The original incident source and the team responsible for detecting the issue are not always clearly identifiable.
- Reassignment history makes it difficult to distinguish incident creation effort from subsequent investigation and resolution effort.
- Reporting may not accurately highlight problematic applications, domains, or operational areas.
- Team-level workload, recurring issue patterns, and opportunities for operational improvement become harder to measure.

Questions and requirements:

1. What is the recommended approach for capturing the original incident source, creating team, and creating operator separately from the Requested By/Caller and Assigned To fields?
2. Can incidents created from Control-M server pages, dashboards, or other portals automatically capture the source system, originating team, operator ID, job name, and relevant application or domain?
3. How can we preserve the original creating team and operator details even when incidents are reassigned multiple times?
4. Can ServiceNow reports or dashboards show the number of incidents created by each team, irrespective of the current assignment group or incident owner?
5. What is the best way to measure incident handling effort, reassignment frequency, resolution time, and the operational areas generating the most incidents?

3. Analysis of Approximately 300 Resolved Incidents Without PRB References

**Problem statement:**

We have observed approximately 300 incidents that were resolved or closed without any associated PRB reference. We need to determine whether these incidents represent genuine one-time occurrences or whether recurring problems have gone unidentified.

Does a high volume of incidents without PRB references indicate that the platform is experiencing stability issues, or could these incidents be routine operational events that do not require formal problem records?

Impact:

- Recurring issues may remain undetected when incidents are reviewed in isolation.
- Potential platform stability concerns may go unnoticed without trend analysis.
- Development teams may lack visibility into patterns that warrant root cause investigation.
- Operational teams may repeatedly resolve symptoms without identifying opportunities for permanent fixes.

Questions and requirements:

1. How can we analyze historical incidents to identify recurring issues based on error messages, incident descriptions, affected applications, Control-M job names, failure codes, and other common attributes?
2. Is there a feasible way to analyze these approximately 300 incidents and classify them into genuine one-time incidents, recurring issues, known errors, and potential problem candidates?
3. What criteria should determine whether an incident requires a new PRB, should be linked to an existing PRB, or can be closed without problem management involvement?
4. How can incident trends, recurrence rates, and failure patterns be used to assess platform stability and identify problematic applications or processes?
5. What reports or dashboards would help track recurring failures, unresolved root causes, and incidents that repeatedly occur without an associated PRB?

4. Automating Incident Creation and Enriching Tickets with Control-M Output

Another area where we see an opportunity for improvement is incident enrichment and automation.

Currently, L1 operators monitor Control-M jobs through server pages and investigate failures by reviewing job output or SYSOUT logs. When creating incidents, operators may need to manually copy relevant error details into the ticket.

Questions and requirements:

1. Can ServiceNow incidents be automatically populated with relevant error messages and diagnostic details extracted from Control-M job output or SYSOUT?
2. Can the integration capture attributes such as job name, folder/application, order ID, run date and time, failure code, relevant log excerpts, and execution status?
3. What integration methods are recommended—for example, native integrations, APIs, webhooks, middleware, or custom automation?
4. Can automated incident creation and enrichment reduce manual effort while preventing duplicate tickets for the same job failure?
5. Can the extracted error details be used to identify recurring failures and associate incidents with existing PRBs?

5. Expected Outcomes and Recommendations

We would like to explore feasible solutions using existing ServiceNow and Control-M capabilities wherever possible, with custom development considered only when necessary.

Our key objectives are to:

- Improve consistency in PRB creation and incident-to-PRB association.
- Identify recurring issues automatically and recommend appropriate problem records.
- Improve incident traceability by preserving the originating team, operator, and source system.
- Generate meaningful reports on incidents and PRBs by team, application, domain, recurrence, and priority.
- Analyze historical incidents to distinguish one-time occurrences from recurring or systemic problems.
- Automate incident creation and enrichment using Control-M job output and error details.
- Improve root cause analysis, permanent resolution, and closure tracking.
- Reduce repetitive manual effort for L1 operators while retaining appropriate human review and approval controls.

I would appreciate recommendations on existing ServiceNow features, Control-M integration capabilities, implementation approaches, or proven industry practices that could address these challenges.**

Thank you in advance for your suggestions and insights!

[abbasshaik4]

Hello @swathik5566,

Please refer to the link below:
Problem Management in ServiceNow: Aligning Inciden... - ServiceNow Community

If it is helpful, please mark it as helpful and accept the correct solution by referring to this solution in the future it will be helpful to them.

Thanks & Regards,

Abbas Shaik