Working with client test runners
Summarize
Summary of Working with client test runners
The client test runner is a browser-based interface in ServiceNow's Automated Test Framework (ATF) that executes automated tests involving UI elements like forms. It supports two types of runners: one for manually started tests and another for scheduled tests. The client test runner operates within the current browser session and runs tests as the logged-in user unless an impersonation step is included.
Show less
Key Features
- Test Execution Control: The test execution property must be enabled to use the client test runner, which is disabled by default to prevent accidental test runs in production environments. Tests should only run on development or non-production instances to avoid data corruption or outages.
- Idle Monitoring and Execution: When idle, the client test runner checks every five seconds for any waiting tests to execute, ensuring tests start as soon as a properly configured client is available.
- Multiple Test Execution: The system now supports running multiple tests simultaneously within the client test runner.
- Debugging Support: Enabling additional debugging functionality adds a Debug Info tab alongside the Execution Frame, showing detailed debug data and capturing screenshots of tests, which are saved to the test result records.
- Browser Recommendations: To maintain performance and avoid memory issues or CPU throttling, run the client test runner in its own browser window, keep it partially visible, avoid screen locking, and maintain browser zoom at 100%. Browsers like Internet Explorer, Edge, and older Firefox versions may require periodic restarts.
- Scheduled Suite Requirements: For scheduled test runs, especially on Mac OS X using Chrome or Safari, running the client test runner in a virtual machine where the screen remains unlocked improves performance and prevents test timeouts. The browser used must match the criteria set in the Scheduled suite run record, and a matching client test runner must be available at the scheduled time.
- JavaScript Command Handling: The client test runner intercepts window commands such as console.log, alert, confirm, and prompt. It automatically responds with default values (e.g., true for confirm, "test value" for prompt) and ignores alerts.
- Active Test Runners Management: Active client test runners are tracked in the Active Test Runners table, accessible via modules filtered for manual or scheduled runners. This table helps monitor runner availability and browser details, especially useful for scheduled suites. Runner status updates are transient and configurable via ATF properties.
Practical Implications for ServiceNow Customers
To effectively use client test runners, ensure test execution is enabled only in safe environments and follow browser best practices to maintain stability and accurate test results. For scheduled automated tests, verify that client test runners meet specified browser criteria and remain active at scheduled times. Utilize debugging features to capture detailed execution info and screenshots for troubleshooting. Monitoring active runners helps maintain visibility and control over test execution activities.
If an automated test includes steps that involve a form or any other user-interface (UI) element, it runs those steps in a browser tab or window called a test runner or client test runner.
The Automated Test Framework supports two types of client test runners: Client Test Runners for manually started tests and Scheduled Client Test Runners for tests started by a schedule.
When test execution is enabled, clicking the Client Test Runner module opens the client test runner in the current browser session. If tests are waiting to be run, the Client Test Runner runs a waiting test. If no test is running, the message Waiting for a test to run displays in the client test runner.
When the client runner is active, it displays the activity of the currently running test in the Execution Frame.
Test execution property
To work with the client test runner module, the test execution property must be enabled.If the test execution property is disabled when you select this module, the system displays a message and a link to the automated test framework properties page where you can enable it.
Additional debugging functionality
If you have enabled additional debugging functionality, the client test runner module displays two tabs: Execution Frame and Debug Info. The Execution Frame displays the information normally shown by the client test runner and the Debug Info displays additional debugging information.
The system takes screen shots from the tests in the Execution Frame tab and records them to the test result record.
Browser recommendations for all tests and suites
- Some browsers have memory-management limitations that make it necessary to occasionally close and restart the browser when running the client test runner. These browsers include Internet Explorer, Edge, and older versions of Firefox. How often you should close the browser depends on the memory allocation in the browser application.
- Some browsers have features that throttle CPU time. To avoid problems, follow these
guidelines:
- Run the client test runner in its own browser window.
- Keep the client test runner at least partially visible on the screen.
- Make certain the screen is not locked or shut off.
- The client test runner takes screen shots as the tests run. For best results with screen shots, leave the browser zoom level set to 100%.
Browser recommendations for scheduled suites
The client test runners for scheduled suites have additional browser requirements.
- On OS X with the client test runner on Chrome or Safari: If the screen is locked or the client test runner tab is not shown, when the system attempts to run the test suite, tests run significantly slower and may time out. For best performance, run client test runners for scheduled suites in a virtual machine (VM) environment in which the screen does not become locked or disabled.
- The browser must meet the criteria you specified on the Scheduled suite run record.
- A client test runner meeting the criteria you specified on the Scheduled suite run record must be available to run the test suite at the scheduled time. The system cannot automatically open a client test-runner session.
Javascript window command intercepts
The Client Test Runner captures window object commands including console.log, console.error, alert, confirm, and prompt, with default responses where necessary.
- Any script that calls window.confirm receives a boolean response of true.
- Any script that calls window.prompt receives the string response test value.
- Any script call to alert is ignored.
Active Test Runners table
When you start a client test runner, the system registers that runner in the Active Test Runners table. You can view this table in the Active Manual Test Runners module and the Active Scheduled Test Runners module. These two modules provide views of the same table, filtered to show only manual or only scheduled test runners.
The Active Scheduled Test Runner module is useful when you create a scheduled suite run. For scheduled suite runs, you can specify the browser to use. To determine the name and version of a browser you want to use, start a scheduled test runner with that browser, then inspect that runner's record in the Active Scheduled Test Runners module.
The data in this table is transient. While the runner is active, it reports in to the system at a specified interval. If the runner does not report in at the expected time, the system marks the runner as inactive. After a period of time the system deletes the runner. You can modify these intervals on the Automated Test Framework properties page.