Automated Test Framework Part 3: Building Blocks

drjohnchun · ‎12-15-2016

Last time, we ran a demo test called "Basic UI Test" and created a Service Catalog Task record. We also saw that, by design, the record was permanently deleted after the test was completed, with the Automated Test Framework (ATF) "automatically taking care of rolling back changes after testing". Before going further with more tests, I'd like to take a deeper look at the various building blocks of ATF both on the surface and behind the scenes; this will give us the lay of the land and help us what to look for later as we create and run more tests.

APPLICATION MODULES

As noted in Part 1, the Navigator provides the following modules for the atf_test_admin role under the Automated Test Framework application menu:

Below is a quick rundown of what you get for each module:

Tests

This module shows list of Tests. Tests include both UI and server tests.

Suites

This module shows list of Test Suites. Test Suites are made up of one or more Tests and/or Test Suites.

Test Results

This module shows list of Test Results. There may be multiple Test Results for a Test.

Suite Results

This module shows list of Test Suite Results. There may be multiple Test Suite Results for a Test Suite.

Run > Client Test Runner

This module opens Clint Test Runner window. This window, labeled as UI Test Runner, may also be opened from the Run Test dialog box. Without this window opened, UI tests won't run.

Run > Test Run Queue

This module shows list of Tests that are Waiting or Running. When running a Test Suite, this lists all Tests that are part of the Test Suite and its child Test Suites, if any. This list doesn't show the execution order, limiting its usefulness. NOTE: Tests can't be scheduled to run at a later time.

Occasionally, when I click on this module while Tests are running, I get the message "Security constraints prevent access to requested page" as shown in the screenshot below:

Run > Suite Run Queue

This module shows list of Test Suites that have Started or are Running. As in Test Run Queue, this list doesn't show the execution order. NOTE: Test Suites can't be scheduled to run at a later time

Administration > Properties

This module shows property settings. As noted in Part 1, these settings are unchecked by default and the first checkbox must be checked to be able to run tests.

Administration > Step Configurations

This module shows list of Test Step Configurations. These are used to build Test Steps. ATF comes with several predefined Test Step Configs and new ones can also be created. Test Steps can only be created from an existing Test Step Config.

Administration > Step Environments

This module shows list of Test Step Environments. These are used in Test Step Configurations and there are two predefined environments: UI and Server. NOTE: this doesn't allow selecting a server instance, for example QA, DEV, etc.

Administration > Test Templates

This module shows list of Test Templates. These are used to build Test Steps in a Test. One Test Template comes with the demo data. A Test Template contains a list of Test Step Configurations in a Glide List.

Administration > Step Configuration Categories

This module shows list of Test Step Configuration Categories. These are used in the Add Test Step dialog box to filter Test Step Configurations when building Test Steps. There are two predefined categories: Form and Server.

TEST HIERARCHY

Using Test Suites, multiple Tests can be bundled. A Test Suite may contain Tests and/or other Test Suites. A Test may belong to more than one Test Suite, as shown in the below hierarchy, whereas a Test Suite may belong to only one parent Test Suite:

When Test Suite A is executed, here's what happens (within the same level in the hierarchy diagram, assume Tests on the left have lower Execution Order, so executed first):

Test 1 runs and finishes
Test 2 runs and finishes
Test Suite B starts
Test 3 runs and finishes
Test 2 doesn't run again, since it already ran in Test Suite A
Test Suite B finishes
Test Suite A finishes

The test sequence is shown in the Run Test Suite dialog box while the Test Suite is running. When a Test Suite has both Tests and Test Suites, like Test Suite A above, the Tests are always executed first before Test Suites.

While ATF allows a Test to be used in multiple Test Suites, care must be exercised when there are dependencies. In the above example, Test 2 was used twice, first under Test Suite A and second time under Test Suite B. We noticed that Test 2 didn't run again under Test Suite B because it already ran under Test Suite A. If Test 2 had dependency on Test 3 in Test Suite B (e.g., using an output value from Test 3), it may not run correctly.

TABLE RELATIONSHIPS

I used GQL Pad to inspect the database and put together the below ERD (Entity Relationship Diagram) showing the various tables used by ATF and their relationships. This, in conjunction with the module descriptions and test hierarchy above, provides an insight into the inner connections in ATF. For clarity, instead of showing all fields, only the reference fields are shown here to highlight the various relationships. We'll reference this later when we discuss how test records are connected and investigate any issues.

I noticed that not all tables had data after loading the demo data; we'll see how and if they're being used as we run more tests later. The Test Suite Test [sys_atf_test_suite_test] table is a many-to-many (m2m) join table that connects between Test Suites and Tests, as shown earlier in Test Hierarchy. Test Template [sys_atf_test_template] has a Glide List for Test Step Configs and is not explicitly related to any tables.

Next time, we'll resume running more Tests as well as creating new ones.

Please feel free to connect, follow, post feedback / questions / comments, share, like, bookmark, endorse.

John Chun, PhD PMP

Winner of November 2016 Members' Choice Award

drjohnchun · ‎03-29-2017

Hi Todd - there are a lot of hoops to jump through to get this working; there must be a good reason for that, so please proceed with caution at your own risk. You can create your own Test Step Config by cloning an OOB record using a Background Script similar to below:

var grOld = new GlideRecord('sys_atf_step_config');

grOld.get('d8fdf5e10b1022009cfdc71437673adc');   // OOB "UI Action Visibility"

var grNew = new GlideRecord('sys_atf_step_config');

grNew.newRecord();

for (var f in grOld) if (f != 'sys_id') grNew[f] = grOld[f];

grNew.name += ' Clone';

grNew.sys_policy = 'NULL';   // make it Unprotected

grNew.setWorkflow(false);   // don't run Business Rule "Check step environment"

grNew.insert();

Please note that this doesn't clone Related Lists. Also, when you try to update this in the form after making changes, the "Check step environment" business rule will prevent your from doing so, which means you can update the record only using Background Script with .setWorkflow(false).

drjohnchun · ‎04-23-2017

Hi Todd - I recently learned that the inability to create custom Form Step Configurations is by design, as I suspected. The reason I was told is because form/UI/browser tests tend to be "brittle", that is, they tend to break easily with changes in UI (by switching to a new UI, for example).

tstocking · ‎04-24-2017

Thank you for the follow up John!

Richard101 · ‎04-25-2017

Is there support in the ATF for unhappy path testing? i.e. make sure a message appears when you try to submit a form with an invalid value.

drjohnchun · ‎04-26-2017

As mentioned in the above thread, UI tests, including "unhappy path" or negative tests, are limited to the available OOB Form Step Configurations, since we can't create our own. In Istanbul, I'm not aware of a way to test what you're describing.

JohnG3 · ‎08-23-2017

Hi Sagar,

Any chance you created the Test Step Configuration for creation of a new record from a Related List?

Use Case: On the Problem Form, click the "New" UI Action on the Problem Tasks Related List

Thank you,

John

praveen_pvs · ‎09-08-2017

Thanks for this nice blog John Chun.

Can you please share how ServiceNow rolls back all the changes it made after executing tests? I.e., Data Cleanup. (e.g. cleaning up of incidents/tasks etc which are created as part of test)

Have got two important considerations before executing tests.

1. Does SN rolls back only the data that was created during the tests? Or rolls back all the data created in that session?

2. What if another user creates some legitimate data (in another session) at the same time while tests are running? Is this data also going to be rolled back?

3. When does the roll back happens? Is it after each test run or when the full test suite completes execution?

4. Is there any mechanism to create and restore save points before and after executing tests?

5. Is there any guidance/recommendation on setting up of large data before running a large test suite?

ATF in Jakarta unleashes a lot of ability to automate tests, but while considering automation at a medium to large scale, need to consider all the above points before going through this route. I have been using Selenium to automate tests, but this ATF seems to be much quicker and easier for any user to automate.

Thanks for your time,

-Praveen

gayatrimuthina · ‎07-30-2018

Can we test UI Pages using ATF.

Mahesh114 · ‎03-19-2020

can we set sequnence of execution in multiple child test suites which are at same hierarchy level in a parent test suite.