ATF Test steps broken after import via Update Set

[Hoppertron]

On our Dev instance, we created a series of ATF Test Suites and Tests which contained Record Validation steps to validate settings in the sys_properties table. (See attached ATF1.png for an example)

These all worked and returned a 100% success ratio on the Dev instance.

We exported the ATF Suites (including all Tests and Steps) in an Update Set and imported them into our Test instance, where we ran the tests again.

About 5 of 14 the tests returned failures, and on closer inspection we noticed that for every failed test, the 'Record' field on the Test Step form was empty.

i.e. the Test Step was missing an input variable: the name of the system property to validate. (See attached ATF2.png)

All other fields (e.g. 'Value' and criteria, table name etc) were intact.

We've not identified a reason for this yet:

• initial suspicion was that it only affected drop-down fields/related lists, but not all drop-down fields are affected and we have observed this happening to string/text fields
• Update set logs show no errors
• No syslog errors are present on the Dev instance when creating the Update Set
• No syslog errors are present on the Test instance when applying the Update Set.
• Both instances are running Jakarta Patch 5.

Has anyone else experienced this happening?   Any ideas for a solution?

(I have searched the Docs site, community, Knowledge base, and Known Errors for ATF in the Hi portal, and haven't found anything).

[Hoppertron]

I've been meaning to update this post with my solution but haven't had time yet.

We spoke to SNC Support and they confirmed that when an ATF test for Record Validation is exported via an Update Set, it references the sys_id of the record that you're checking and NOT the filter criteria that you used to find the record.   Therefore, it will not work on another instance unless the target instance was cloned from the source.   This is "expected behaviour", not a bug.

To work around this I produced a server-side script to run the record validations based on record name-value pairs instead of sys_id.   The script is called by an ATF Test Step, and returns an overall success or failure for the Step, with a detailed log of which tests passed and which failed stored within the step.

Will share if I can.

[Juliano Tripodi]

Hi,

Are you able to share the script?

Tks !!

[joel_fischer]

For anyone else reading this community post.

The issue is if your test step directly references a record by sys_id, and that record is not on the instance to where you've moved your test, then the test will not find that record. This is expected, but not an optimal experience.

To get around relying on sys_ids, you can use Record Query to find a record by sys_properties.name, then pass the query result (first record) to the next step.

Pass the Record Query step output to Record Validation step input

https://docs.servicenow.com/bundle/kingston-application-development/page/administer/auto-test-framework/task/atf-retrieve-value.html

[mlee]

Joel,

You are correct! If you have a lot of reference values in your test, which I do, this doubles the number of steps required to execute the test, increase the time to build a test and increase the time to run the tests.

For me, I do xml exports of depending values and import those into each environment so the sys_id remains the same across the environments. This approach only works if you are setting up new values/variables/references.

Mon

[eagleman]

We are experiencing the same problem in Kingston now patch 5. 
is not about reference to existing real records. Simply all the steps of some test cases are incomplete.

They have the full description populated, steps are there but when you get into them they are empty, no table selected, no fields selected, nothing.
Happends around 50% of test cases moved from dev to test....