ServiceNow RoboArchitect
An Automated Application Architecture Framework
ABSTRACT
This article endeavors to establish an automated application architecture framework for an entire ServiceNow instance, leveraging the existing metadata within the ServiceNow environment. It focuses on the package as the fundamental modular entity on the Now Platform, illustrating methods for metadata extraction and subsequent data analysis through a data science lens. The analysis is presented via a Jupyter notebook, to facilitate empirical validation and extend the scope for further exploration.
PROBLEM STATEMENT
Within the context of ServiceNow architecture, packages emerge as the quintessential modular entities, residing in the sys_package table. A package may manifest as either a plugin or a scoped application, with further distinctions between store apps and custom apps.
The hierarchical structure of the sys_package table is depicted in the Figure 1.
This study concentrates on delineating specific package dependencies in ServiceNow, aiming to elucidate the interdependencies among packages. Employing a LEGO™ analogy, packages serve as the foundational building blocks of instance capabilities, where primary packages underpin the functionalities of higher-level packages (Figure 2). This structure permits users to develop their applications atop this modular hierarchy.
Each package’s architectural data is cataloged in the Application Files table, referred to as "metadata" (originally termed sys_metadata).
Packages comprise metadata, each contributing capabilities to and utilizing capabilities from other packages (termed new application file classes, or AFCs, and application files, or AFs of those AFCs, respectively). AFCs function similarly to the studs on LEGO™ blocks, facilitating connections and constructing a hierarchical framework, as illustrated in Figure 3.
To exemplify this overarching software architecture concept:
- The "Environmental, Social, Governance Management" package utilizes the Flow Designer and incorporates objects of the sys_hub_flow class, supplied by the "Flow Designer Model" package.
- Any package incorporating a business rule utilizes the sys_script class provided by the foundational "System" package (apps/system1).
DATA PROCESSING
Mapping the ServiceNow software architecture presents a compelling data science challenge, following a classical data pipeline shown in Figure 4:
- Data Collection: Generating and exporting datasets from the target ServiceNow instance.
- Data Ingestion: Loading data into the processing environment.
- Data Preparation: Organizing and cleansing raw data in preparation for analysis.
- Data Analysis: Deriving insights and outcomes from the data, employing assumptions and approximations.
- Data Visualization: Creating clear and impactful visualizations to convey research findings.
The artifacts of this study are accessible via a GitHub repository, which includes:
- Extracted datasets from the ServiceNow target instance and the associated extraction scripts demonstrating step 1.
- A computational notebook demonstrating steps 2-5.
An overview of key findings from each phase is provided below.
1. Data Collection
Three scheduled jobs were established to extract the sys_db_object, sys_package, and sys_metadata tables as CSV attachments.
The dataset comprises files of the following sizes:
- sys_package: 156 KB
- sys_db_object: 636 KB
- sys_metadata: 49 MB
The following scripts were used:
//Export packages
gr = new GlideRecord("sys_package") gr.query() gs.info('"name","sys_id"') while(gr.next()) { gs.info('"' + gr.name + '","' + gr.sys_id + '"') }
//Export tables
gr = new GlideRecord("sys_db_object") gr.query() gs.info('"name","super_class","sys_package"') while(gr.next()) { var sys_id = gr.getValue("super_class") if (sys_id) { x = new GlideRecord("sys_db_object") x.get(sys_id) y = x.getValue("name") } else { y = "" } gs.info('"' + gr.name + '","' + y + '","' + gr.sys_package + '"') }
//Export metadata
var batchSize = 100000; var startIndex = 0; var gr = new GlideRecord('sys_metadata'); while (true) { gr.chooseWindow(startIndex, startIndex + batchSize); gr.query(); if (!gr.hasNext()) { break; // Exit the loop if there are no more records } while (gr.next()) { data.push('"' + gr.sys_class_name + '","' + gr.sys_package + '"'); } startIndex += batchSize; // Move to the next batch } gs.info("Number of records: " + data.length) var attachment = new GlideSysAttachment(); var fileName = "sys_metadata_export.csv"; var contentType = "text/csv"; try { attachment.write(current, fileName, contentType, data.join('\n')); } catch(e) { gs.info("Error " + JSON.stringify(e)) }
2. Data Ingestion
The three files were imported into a Jupyter notebook as Pandas [1] dataframes, serving as the foundation for Python-based data analysis. A Pandas dataframe functions akin to an Excel spreadsheet but equipped with extensive data analysis capabilities. Google Colaboratory was utilized for the Jupyter notebook environment.
3. Data Preparation
Data preparation involved employing Pandas to consolidate the three dataframes into a singular dataframe, listing unique packages alongside the AFCs provided and consumed by each. A Directed Graph (DiGraph) was then instantiated using NetworkX [2], a premier library for network graph analysis in Python. The resulting NetworkX graph comprised 1,766 nodes (packages) and 14,285 links (relationships between packages, delineated by an AFC provided by the source and consumed by the target).
4. Data Analysis
Analysis of the NetworkX graph revealed that 1,442 nodes were terminal, either not providing any AFC or their provided AFCs were not utilized by any package within the environment. The refined graph, excluding terminal nodes and links, consisted of 324 nodes and 4,611 links. Notably, the graph contained cycles, indicating it was not a Directed Acyclic Graph (DAG). This was unexpected, as a structured application architecture typically exhibits layering. However, the graph was nearly acyclic, with 19 bidirectional cycles identified, where each package in a pair consumed AFCs from the other.
Figure 5. 2-Cycles as reciprocal Application File Classes consumers and providers.
This reciprocal dependency, possibly resulting from packages generating new AFs post-installation, warranted further exploration. A heuristic approach was adopted to eliminate links within these bidirectional cycles, leaving 19 cycles with more than two nodes. These remaining cycles were analyzed to determine feasible link removals to break the cycles.
Subsequently, the graph, now comprising 324 nodes and 4,568 edges, was restructured into a directed acyclic format, allowing for layered organization. The NetworkX function nx.topological_generations(G) was employed to assign a layer to each node (package), resulting in a diagram with 23 levels. This layering dictates that packages at level Ln rely solely on application files from packages at levels L0 through Ln-1.
5. Data Visualization
Visualization efforts utilized Plotly [3] and IPyWidgets for interactive components. Although a classic DAG-style visualization was initially attempted, the complexity and size of the graph necessitated an alternative approach. An architectural-style visualization displaying all 23 architecture levels were produced (see Figure 6). Click on the following link for an high-resolution file: https://github.com/nicola-attico/roboarchitect/blob/main/layered_architecture.png
The notebooks enable users to select any package via an IPyWidget dropdown and highlight (in red) the packages that provide capabilities to the selected package, as shown in Figure 7 (click here for an high-res, zoomable version: https://github.com/nicola-attico/roboarchitect/blob/main/layered%20_architecture_with_dependencies.p...) . This feature accommodates both out-of-the-box and custom packages, provided they are present in the analyzed environment.
Observations on the resultant architecture include:
- Level 0 features AFs unassigned to any package, representing core capabilities (e.g., 'sys_dictionary_override', 'sys_glide_object').
- Levels 1 and 2 house foundational packages for the Now Platform, such as various System packages.
- Higher levels introduce more advanced packages, building upon the functionalities of preceding layers.
- Final packages, not depicted in the visualization, are conceptually positioned at level 24 but can still be explored via the dropdown selection.
CONCLUSIONS & NEXT STEPS
This article and accompanying notebook constitute an early effort to automate the mapping of ServiceNow architecture using available metadata. It invites scrutiny of the notebook with the provided data, aspiring to develop a tailored architectural model.
While application file provision and consumption are critical, they represent only one facet of technical dependencies. A comprehensive architecture must encompass the full spectrum of application metadata. A complete sys_metadata table export from the same environment, amounting to 3.17 GB, is feasible. Additional dependencies, such as Business Rules operating on tables provided by other packages or table extensions and references, merit consideration. Further analysis, incorporating all potential relationship types, is underway through an additional notebook.
REFERENCES
[1] The pandas development team. (2024). pandas-dev/pandas: Pandas (v2.2.0). Zenodo. https://doi.org/10.5281/zenodo.10537285
[2] Aric A. Hagberg, Daniel A. Schult and Pieter J. Swart, “Exploring network structure, dynamics, and function using NetworkX”, in Proceedings of the 7th Python in Science Conference (SciPy2008), Gäel Varoquaux, Travis Vaught, and Jarrod Millman (Eds), (Pasadena, CA USA), pp. 11–15, Aug 2008
[3] Plotly Technologies Inc., Collaborative data science, Plotly Technologies Inc., Montréal, QC, 2015 URL: https://plot.ly
