Why do Test(s) complete successfully individually, but fail when put together in a Test Suite?

[guislainlagron]

Hi,

I have created 5 Tests, which complete successfully when run individually, but 3 of them fail when run in a Test Suite.

Specifically, the 3 which fail (all fail with the same pattern), deep into the Test by failing on the same "Update" UI Action:

even though the previous "Set Fields Value" step completes successfully.

because it can't find the record it just updated...

NOTE: Each of these 5 Tests run successfully individually.

Any hint is well appreciated.

Guislain

[tony_barratt]

Hi Guislain,

I expect you are on the right track by taking a good look at why the first failure occurred.

Maybe you could find out what javascript error was generated?

Considering the issue you mention from a high level may also be helpful - there are some quite interesting suggestions here:

unit testing - Test Method Fails in Test Suite but passes individually in .net C# - Stack Overflow

Best Regards

Tony

[guislainlagron]

Hi Anthony,

Thanks for the reference. Perhaps these two additional tips may shed some light.

1. Even when I run the Test individually (i.e. not in a Test Suite), when I change who I impersonate earlier in the script, then usually the same Update UI action step fails with the same "record not found" error, even if I run the Test individually (vs in a Suite). If I update the step again - deselecting the Update action and reselecting it - then the UI action completes successfully when I run the Test individually.
2. Several UI actions (incl. Update, Delete) actually have multiple options for the same action, and I have no way to distinguish the difference between the two.

                 

It feels that these two additional tidbits have a correlation relationship with the behavior, albeit perhaps not a causation relationship.

Do you know more about the multiple options for the same value?

Thanks,

Guislain

[guislainlagron]

Hi Anthony,

Getting closer...

So I was able to distinguish each of the Update actions, but it requires selecting each and getting information about each,

       

so I made sure that I selected all Update actions which corresponds to the sysverb_update verb, since it is supposed to return to the previous screen.

       

I also decomposed each Test so that they would impersonate only one role.

Then I started to run each Test individually again, and I did get one Test to execute inconsistently, Specifically it failed

       

with the same error

       

               

       

       

only to complete successfully, just by rerunning it again UNCHANGED!!!

In other the words, the behavior observed in a Test Suite can also occur sporadically when running the same Test several times.

It leads me to think that session information is not cleaned up correctly after each Test, so:

• How do I clean session information or force a new session (I was hoping that Impersonating a User would that)?
• How do I close a form without Submitting it (which either uses Save or Submit by default)?

Thanks for any tips,

Guislain

[guislainlagron · Accepted solution]

FYI.

I will ignore the 3 false positive Tests (despite assertion steps early in each such test to verify expected state of test data) in my Test Suite comprising of 32 Tests, then rerun those 3 failed Tests which complete successfully.

PS: I also noticed 2 other strange behaviors, which I will cover in separate posts.

Guislain