How to setup ATF for CI/CD pipeline

[JC S_]

We've been trying to work on CI/CD for our ServiceNow development and one key part of it is the Automated Test Framework. But almost on all resources released by ServiceNow about CI/CD pipeline, they all gloss over the ATF test suite part. Seems like on all the sample video they provided, the test suite does not contain any real tests that's why they always "pass".

Recent videos released by ServiceNow for CI/CD pipeline setup:

• Setting up your first CI/CD pipeline with GitHub Actions
• Setting up your first CI/CD pipeline with Azure Pipelines
• Setting up your first CI/CD pipeline with GitLab CI/CD Pipelines
• Setting up your first CI/CD pipeline with Jenkins

We've already tried calling an ATF test suite (with client actions) on our pipeline and all we get is the test suite getting skipped because there's no matching client test runner found. From what we gather there is a docker image referenced in the sample pipeline YML template and that should have been the basis of a docker container that will be running the identified ATF test suite.

Is there any special setup that we need to do to ensure that our ATF test suites are run properly as part of the CI/CD pipeline?

[JC S_]

@Brad Tilton @Chiarng Lin @Chiarng Lin any insights on this?

[Chiarng Lin]

The videos demonstrating first time setup for CI/CD pipelines was intended to show how to get going quickly, and so you are correct that the ATF Test Suite I built is a trivial example that doesn't include UI tests. 

We're addressing that pain point as a part of our Rome release - having to keep a browser window/tab open on a Scheduled Client Test Runner in order to run ATF Test Suites with UI tests. 

Unfortunately for now, the only workaround is to keep a browser window/tab open somewhere. 

[JC S_]

Thanks for your prompt feedback here.

Just to clarify further, what's the use of the docker image involved in the pipeline template? I was under the assumption that the pipeline will trigger a container that is able to run the client test of the ATF.

I highly recommend that the ATF shortcomings should not be glossed over on your external communications and be addressed directly because automated testing is a key part of CI/CD. Transparency will be very much appreciated so we can align our company's efforts regarding CI/CD on ServiceNow.

Best if ServiceNow can provide guidance on how we can address it now given the current limitations of the product while we are waiting months away the release where this is going to be addressed.

[Venkata1]

@Chiarng Lin  - I am looking for help on implementing same in our company but your script refers to task.sh file which is not available in gitlab project. 

https://gitlab.com/ServiceNow-DevX/sncicd-gitlab-pipeline

or 

https://gitlab.com/service-now-dev/extension/-/blob/master/README.md#user-content-required-parameters

Appreciate if you can guide us here as many people in community looking for same steps.