Automated Test Framework Part 3: Building Blocks

[drjohnchun]

Last time, we ran a demo test called "Basic UI Test" and created a Service Catalog Task record. We also saw that, by design, the record was permanently deleted after the test was completed, with the Automated Test Framework (ATF) "automatically taking care of rolling back changes after testing". Before going further with more tests, I'd like to take a deeper look at the various building blocks of ATF both on the surface and behind the scenes; this will give us the lay of the land and help us what to look for later as we create and run more tests.

APPLICATION MODULES

As noted in Part 1, the Navigator provides the following modules for the atf_test_admin role under the Automated Test Framework application menu:

Below is a quick rundown of what you get for each module:

Tests

This module shows list of Tests. Tests include both UI and server tests.

Suites

This module shows list of Test Suites. Test Suites are made up of one or more Tests and/or Test Suites.

Test Results

This module shows list of Test Results. There may be multiple Test Results for a Test.

Suite Results

This module shows list of Test Suite Results. There may be multiple Test Suite Results for a Test Suite.

Run > Client Test Runner

This module opens Clint Test Runner window. This window, labeled as UI Test Runner, may also be opened from the Run Test dialog box. Without this window opened, UI tests won't run.

Run > Test Run Queue

This module shows list of Tests that are Waiting or Running. When running a Test Suite, this lists all Tests that are part of the Test Suite and its child Test Suites, if any. This list doesn't show the execution order, limiting its usefulness. NOTE: Tests can't be scheduled to run at a later time.

Occasionally, when I click on this module while Tests are running, I get the message "Security constraints prevent access to requested page" as shown in the screenshot below:

Run > Suite Run Queue

This module shows list of Test Suites that have Started or are Running. As in Test Run Queue, this list doesn't show the execution order. NOTE: Test Suites can't be scheduled to run at a later time

Administration > Properties

This module shows property settings. As noted in Part 1, these settings are unchecked by default and the first checkbox must be checked to be able to run tests.

Administration > Step Configurations

This module shows list of Test Step Configurations. These are used to build Test Steps. ATF comes with several predefined Test Step Configs and new ones can also be created. Test Steps can only be created from an existing Test Step Config.

Administration > Step Environments

This module shows list of Test Step Environments. These are used in Test Step Configurations and there are two predefined environments: UI and Server. NOTE: this doesn't allow selecting a server instance, for example QA, DEV, etc.

Administration > Test Templates

This module shows list of Test Templates. These are used to build Test Steps in a Test. One Test Template comes with the demo data. A Test Template contains a list of Test Step Configurations in a Glide List.

Administration > Step Configuration Categories

This module shows list of Test Step Configuration Categories. These are used in the Add Test Step dialog box to filter Test Step Configurations when building Test Steps. There are two predefined categories: Form and Server.

TEST HIERARCHY

Using Test Suites, multiple Tests can be bundled. A Test Suite may contain Tests and/or other Test Suites. A Test may belong to more than one Test Suite, as shown in the below hierarchy, whereas a Test Suite may belong to only one parent Test Suite:

When Test Suite A is executed, here's what happens (within the same level in the hierarchy diagram, assume Tests on the left have lower Execution Order, so executed first):

1. Test 1 runs and finishes
2. Test 2 runs and finishes
3. Test Suite B starts
4. Test 3 runs and finishes
5. Test 2 doesn't run again, since it already ran in Test Suite A
6. Test Suite B finishes
7. Test Suite A finishes

The test sequence is shown in the Run Test Suite dialog box while the Test Suite is running. When a Test Suite has both Tests and Test Suites, like Test Suite A above, the Tests are always executed first before Test Suites.

While ATF allows a Test to be used in multiple Test Suites, care must be exercised when there are dependencies. In the above example, Test 2 was used twice, first under Test Suite A and second time under Test Suite B. We noticed that Test 2 didn't run again under Test Suite B because it already ran under Test Suite A. If Test 2 had dependency on Test 3 in Test Suite B (e.g., using an output value from Test 3), it may not run correctly.

TABLE RELATIONSHIPS

I used GQL Pad to inspect the database and put together the below ERD (Entity Relationship Diagram) showing the various tables used by ATF and their relationships. This, in conjunction with the module descriptions and test hierarchy above, provides an insight into the inner connections in ATF. For clarity, instead of showing all fields, only the reference fields are shown here to highlight the various relationships. We'll reference this later when we discuss how test records are connected and investigate any issues.

I noticed that not all tables had data after loading the demo data; we'll see how and if they're being used as we run more tests later. The Test Suite Test [sys_atf_test_suite_test] table is a many-to-many (m2m) join table that connects between Test Suites and Tests, as shown earlier in Test Hierarchy. Test Template [sys_atf_test_template] has a Glide List for Test Step Configs and is not explicitly related to any tables.

Next time, we'll resume running more Tests as well as creating new ones.

Please feel free to connect, follow, post feedback / questions / comments, share, like, bookmark, endorse.

John Chun, PhD PMP

Winner of November 2016 Members' Choice Award

[drjohnchun]

Here's the series so far:

1. Automated Test Framework Part 1: First Impression
2. Automated Test Framework Part 2: Form Submit
3. Automated Test Framework Part 3: Building Blocks

[sahitya1]

Hi,

Related list validation(Edit/New button visibility onclick() action based on role)

can it be done using ATF?

Thank you.

[drjohnchun]

You'll need to create a Test Step Configuration for that. You can look at some OOB Test Step Configs, for example, UI Action Visibility / Click a UI Action, and make a copy and modify them.

[calicea]

How about creating a load test with ATF?

[Robbie]

Hi John,

I just wanted to give you a shout out and thank you for the feedback and review of the ATF. It's extremely helpful and provides a great overview and hand-on experience of what to expect on initial view.

Thanks for sharing - appreciated.

Robbie

[drjohnchun]

Thanks Robbie - glad to hear that.

John

[drjohnchun]

What kind of load testing do you have in mind? FYI, ATF can run tests only in series (one test at a time).

[tstocking]

Have you figured out a way to determine if a drop-down doesn't contain certain choices? It seems that we can only test for the presence of a value but not the absence of one. For instance, the ability to set the state of an incident to "Closed" is restricted by removing the option through a client script. We would like to test that only the values we expect to see are available but I haven't found a good way to test this through automation.

Also, what about validating field messages?

[drjohnchun]

Let me try to provide some clues to get you going.

Prerequisite: you'll need to have a good understanding of querySelector (aka CSS Selector).

Navigate to Automated Test Framework > Administration > Step Configurations and open "UI Action Visibility". Then inspect Step execution script and scroll to about halfway until you see the following function:

function checkVisibility(actionId) {
   var queryString = "[gsft_id='" + actionId + "']";
   var actionIdExist = testFrameWindow.document.querySelector(queryString);

   if (actionIdExist == null)
       return false;

   if (!isVisible(actionIdExist))
       return false;

   return true;
}

You can copy this to create your own Step Configuration, but with the following:

var valueExist = testFrameWindow.document.querySelector(queryString);

Here, replace "queryString" with the CSS Selector for the "Closed" select option element in the DOM (View Source in the browser will show you what to use). Then you can use it to test the presence or absence of the dropdown value in the field.

You can take a similar approach for testing field messages.

I'd recommend writing a generic function that can test the presence/absence of any DOM element and using it in a reusable Test Step Config.

ps. sorry, I always like using Syntax Highlighting for JavaScript but the editor isn't giving me that option today, somehow. I had to manually insert the <pre> tags to get what you see here.

[tstocking]

Hi John,

Thank you for the response. I was attempting to follow the path you provided however I hit a few road blocks. For starters, the "Step Environment" field will only allow me to select "Server - Independent' and not "UI'. Reviewing the dictionary entry for this field shows that the filter is set to not show the "UI" option. If I remove that filter so that I can select it, I receive an error stating that I need the "maint" role to be able to create a test step with that step environment. I cannot add this role to my user account even with the 'security_admin' role and the 'Elevate Roles' option enabled.

Were you able to copy the "UI Visibility" Step Configuration and modify it accordingly?