DevOps test tool integration

Test tool integration lets you view test results in DevOps for Jenkins, Azure DevOps, GitHub, GitHub Enterprise, and GitLab unit, functional, and performance tests.

For GitLab and Jenkins, only JUnit test type integration is supported.

Note:

For other test types, use the DevOps - POST /devops/tool/{capability} endpoint of the DevOps API.

• Selenium tests run and published using TestNG are reported by the Jenkins plugin for ServiceNow DevOps.
• Test type categorization is supported.
• Additional tests results reported by tools, such as Apache JMeter, can be processed in DevOps using custom Flow Designer subflows (no pipeline changes required).

Category	Test type
Unit	JUnit (default) NUnit XUnit Unit test Note: For GitLab and Jenkins, only the JUnit test type integration is supported. For ADO, GitHub, and GitHub Enterprise, the JUnit, NUnit, XUnit, and Unit test type integrations are supported. You can change the default test type by modifying the [sn_devops.default_test_type] DevOps property.
Functional	Integration Regression Smoke System User Acceptance
Performance	Load

Test type mapping

The test type mapping connects the test type and entity being tested with the DevOps tool (DevOps > Integrations > Test Type Mappings module.)

An accurate test type mapping ensures that the test type always appears as intended in the test summary results.

Field		Description
Test type		JUnit Nunit Xunit Unit test Note: For GitLab and Jenkins, only the JUnit test type integration is supported. For ADO, GitHub, and GitHub Enterprise, the JUnit, NUnit, XUnit, and Unit test type integrations are supported. Integration Regression Smoke System User Acceptance Load
DevOps Entity Id	Table name	DevOps table name that contains the entity linked to the test results (in the test report payload). Step [sn_devops_step] Pipeline [sn_devops_pipeline] Note: Only DevOps Step and Pipeline tables are supported.
	Document	Name of the entity specified in the selected table. For example, the name of the step, pipeline, artifact, or package.
Test File Paths (Jenkins tests only)		Path to the generated test result file on the Jenkins server. This is useful so test reports with attributes that don't conform to JUnit or TestNG implementation, such as JMeter for example, can still be leveraged by DevOps. Separate multiple files by a comma. Note: You must use a Flow Designer subflow to transform a raw test payload.
Tool integration		Tool that's running the test.
DevOps Table		DevOps table that corresponds to the table name in the DevOps Entity Id setting.

Figure 1. DevOps test type mapping

Figure 2. Example yaml file of ADO unit test

Figure 3. Example yaml file of GitHub unit test

Transforming a raw test payload

You can transform a raw test report (a report containing attributes that do not conform to JUnit or TestNG implementation), such as JMeter for example, by configuring the DevOps Test Subflow Policy decision table to call a custom subflow (Decision Tables > Decision Tables module).

Note:

You must create the custom Flow Designer subflow that transforms the raw test payload.

If there is more than one test type in a performance stage, you can use the DevOps Test Type Policy decision table to configure the test type for each test so the test result payloads are transformed correctly.

Figure 4. DevOps decision tables

Table 1. DevOps decision tables

Decision table	Purpose	Configuration
DevOps Test Subflow Policy	To automatically call a custom subflow that transforms the raw payload received by the tool. Decision inputs: Test Result Payload Test Type	Create a decision that specifies the custom subflow to call when the raw payload is received. Set the conditions to contain fields that would be included in the raw payload. For example, to call Jenkins BZ Performance Test custom subflow: Conditions: Test Type is Load (Load is the test type configured for performance tests) Test Result Payload contains throughput Test Result Payload contains concurrency Answer: FLow: Jenkins BZ Performance Test
DevOps Test Type Policy	To automatically set test types when more than one type of test is configured in a performance test stage. This is necessary so the results for the second test type get transformed correctly. For example, when both a Load performance test and a JUnit performance test are mapped in the same DevOps step, the JUnit test results won't get formatted correctly unless a decision is created. Decision inputs: Step Test Result Payload Tool Integration Pipeline	Create a decision for each type of test in the performance test stage to set the test type. Load test: Conditions: Step is Perf Tests Test Result Payload contains throughput Test Result Payload contains concurrency Answer: TestType: Load JUnit test: Conditions: Step is Perf Tests Test Result Payload does not contain throughput Test Result Payload does not contain concurrency Answer: TestType: JUnit

Figure 5. DevOps multiple performance test types

Figure 6. DevOps multiple test type mappings

Figure 7. DevOps decision table decision

Test summary results

You can view test summary results these ways.

• DevOps > Test Results module (Test Summaries and Performance Test Summaries).
• DevOps change request - Test Results related list.
• DevOps Pipeline UI - Quality tile.

Figure 8. DevOps performance test summary example

Expected standard JSON Notification capability payload - Test tool

Functional:

{ 
"name": "CorpSite-selenium#55", 
"duration": 78.802, 
"passedTests": 4, 
"failedTests": 0, 
"skippedTests": 0, 
"blockedTests": 0, 
"totalTests": 4, 
"startTime": "2020-06-30T18:12:31Z", 
"finishTime": "2020-06-30T18:12:31Z", 
"passingPercent": 100, 
 
 
// Use Artifact OR Package OR Build + Stage + PipelineName Attributes 

Send only one Attribute combination. For example, send Attributes of either  Artifact or Package, or the combination of Build + Stage + PipelineName.

If you send more than one Attribute, priority is given in the following order and the low priory one is ignored. For example, if you send attribute for both packages and artifacts, then attribute of package is considered and the attribute of artifacts is ignored.

1.packages
2.artifcats
3.buildNumber + stageName + pipelineName

"packages": [{"name": "CorpSite-pkg1"}], 
"artifacts": [{"name": "CorpSite-artifact", "version": "1.0.0"}], 
"buildNumber": "55", 
"stageName": "test", 
"pipelineName": "CorpSite-selenium", 
} 

Notes:
- The pipelineName attribute value must be same as the value in the Orchestration pipeline field of the Pipeline [sn_devops_pipeline] table.
- The stageName attribute value must be same as the value in the Orchestration stage field of the Step [sn_devops_step] table.

Performance:

{ 
"name": "Performance Tests", 
"url": "http://abc.com", 
"startTime": "2020-06-30T18:12:31Z", 
"finishTime": "2020-06-30T18:12:31Z", 
"duration": 78.802, 
"maximumVirtualUsers": "", 
"throughput": "", 
"maximumTime": "", 
"minimumTime": "", 
"averageTime": "", 
"ninetyPercent": "", 
"standardDeviation": "", 
 
// Use Artifact OR Package OR Build + Stage + PipelineName Attributes 

Send only one Attribute combination. For example, send Attributes of either  Artifact or Package, or the combination of Build + Stage + PipelineName.

If you send more than one Attribute, priority is given in the following order and the low priory one is ignored. For example, if you send attribute for both packages and artifacts, then attribute of package is considered and the attribute of artifacts is ignored.

1.packages
2.artifcats
3.buildNumber + stageName + pipelineName

"packages": [{"name": "CorpSite-pkg1"}], 
"artifacts": [{"name": "CorpSite-artifact", "version": "1.0.0"}], 
"buildNumber": "55", 
"stageName": "test", 
"pipelineName": "CorpSite-Performance", 
} 

Notes:
- The pipelineName attribute value must be same as the value in the Orchestration pipeline field of the Pipeline [sn_devops_pipeline] table.
- The stageName attribute value must be same as the value in the Orchestration stage field of the Step [sn_devops_step] table.

• Configure a test tool in DevOps
  Configure a test tool in DevOps to view unit, functional, and performance test results.
• Add test results to change requests using test API
  Add test results to DevOps change requests using the test capability of the DevOps - POST /devops/tool/{capability} API. The API is available for all test types for various tools connected to DevOps Change Velocity.
• Add attachments to change requests using attachment API
  Add test report xmls as attachments to DevOps change requests using the attachment API.