Question about Automated Test Framework best practices

[bbf3562]

I am new to use ATF and I am using that to test the story with scrum tasks form functionality to make sure the form opened and closed successfully. I came to find that there are two way to test it, one is to use ATF to run query record in background to insert/update data in table or use ATF to impersonate test user, open new form, fill out values, and close the form, and repeat. The big difference between these two is the speed. ATF running in background will complete in seconds vs ATF filling out forms complete in minutes. The cons about running in background query is that I can't monitor the activities in screen and they don't take screenshot for running query in background, only do for filling out forms. For security/validation, I am not sure which one is better. So here are my questions,

1. Which one is the best practice when I need ATF to create new test user and use it to fill out the forms(Story, Request, Incident, Change, etc) to make sure it function correct for before and after the upgrade?
2. What solution do you use where there is a role security issue in different environments when it come to impersonate user? I have 4 instance environments, Development (non-prod), Integration(non-prod), Test(non-prod), and Production. Each environment have different roles level for user, for example I am able to impersonate user who have admin role in development instance but I can't impersonate in other 3 environments. When it come to ATF, that will be a problem when it come to impersonate because it need to impersonate user that have role like a admin that can create test user, add test user to groups, etc.
3. Do you suggest create a permanent test user for ATF to impersonate or it better for ATF to create a temporary test user, assign to group, etc and then impersonate and once the test completed, it delete the test user?

Any suggestion is appreciated!

[Ankush Agrawal · Accepted solution]

Hi

1. If your forms have many client side scripts and you want the test to cover all those, please use the UI step. Otherwise it is always suggested to use server-side scripts.
2. We have created a few common users which we use in the test. This helps to save a lot of time in the creation of user and assigning the right roles. We run the test with a user who has appropriate impersonation role so that the user can impersonate the test users. This approach has only one issue: the test data should not be modified by anyone (e.g: roles, group etc.) otherwise your test will give false results. 

[bbf3562]

Hi Ankush,

Thank you for suggestions! Quick question about your response on #2, do you recommend to create a permanent test users by admin user or temporary test users creating from ATF? What are the pros and cons on both sides?

[Ankush Agrawal]

Glad that you asked about pros and cons. There are not any major differences in terms of maintenance. My favorite is approach 1 below which handles clone and the fastest in execution.

Approach 1: Create the permanent users

Pros:

1. No need to run the create user scripts in each instance post cloning. 
2. sys_id will be consistent across the instances (if you require to use them ever in any of the tests). 
3. If you require to change the configuration of any field on the user record, it has to be done at the source instance and all the tests will be using the same update data. No script change is required.

Cons:

1. If you clone production instances to sub-prods, these can be created in prod and readily available in sub-prods. You have to keep them inactive in prods and activate them in sub-prod via a post clone scripts. 
2. Test data will be lying in the production instance which is not a very good idea. It has to be hidden/masked as per the corporate policy.

Approach 2: Create permanent users via ATF before all the test executions

Pros:

1. No need to worry about the cloning of sub-prods.

Cons:

1. sys_id will differ in each instance
2. Need to run the user creation script once before kicking off the tests
3. If any change in the user field is required, that has to be done on the script and require data cleanup.

Approach 3: Create users via ATF before each the test executions

I guess this is what you are following. You can do one modification here. Create a Step Configuration to create users which will set few mandatory parameters as per your instance requirement and take 2 parameters:

1. Roles
2. Groups

This step configuration can be handily used in all the tests to quickly create users.

Pros:

1. Each test can control the users as per specific requirements
2. Any change in the mandatory fields on user record will require to modify a single step configuration. Every subsequent test execution will be good.

Cons:

1. Longer test execution time
2. To differentiate various users, the user will need to fill a lot of data while using this step.

[bbf3562]

Excellent! Thank you!