I've been having some strange issues with Test Studio, which manifests itself as my tests not working as they should be..
If I run a set of commands using the "Tests" Tab and then executing a test manually, I will get passes all throughout. However, when I add them into a test list to run consecutively, I get the complete opposite. These tests will fail or succeed, and it really seems quite random. I don't understand why this is the case because they work perfectly outside of test lists.
3 Answers, 1 is accepted
0
Hi Dayna,
Thank you for reaching us out.
As of the description I suspect that the speed of execution causes the observed random failures. Probably you would need to insert execution delays or wait steps to ensure the page is loaded and the elements are present when attempting to take action against them.
Please try to modify the tests accordingly and in case you still face troubles you could not determine how to overwhelm share the failure details and preferably the test list results file which could be found in the project root folder under the Results sub-folder.
Thanks in advance for your cooperation!
Regards,
Elena Tsvetkova
Progress Telerik
Thank you for reaching us out.
As of the description I suspect that the speed of execution causes the observed random failures. Probably you would need to insert execution delays or wait steps to ensure the page is loaded and the elements are present when attempting to take action against them.
Please try to modify the tests accordingly and in case you still face troubles you could not determine how to overwhelm share the failure details and preferably the test list results file which could be found in the project root folder under the Results sub-folder.
Thanks in advance for your cooperation!
Regards,
Elena Tsvetkova
Progress Telerik
0
Stéphanie
Top achievements
Rank 1
Rank 1
answered on 03 Jan 2018, 12:37 PM
Hello,
I need a precision to understand what is the difference when the test run alone and when it run in a test list ? I have the same issue like Dayna, some tests work fine alone and fail when in TestList.
I tried your solution and it seem to be ok ! but i want to understand the reason of the difference.
Thank you for your response,
Regards
Stéphanie
I need a precision to understand what is the difference when the test run alone and when it run in a test list ? I have the same issue like Dayna, some tests work fine alone and fail when in TestList.
I tried your solution and it seem to be ok ! but i want to understand the reason of the difference.
Thank you for your response,
Regards
Stéphanie
0
Hello Stéphanie,
Thank you for addressing this forum post.
One of the main tasks in the automation process is to ensure stable test execution. In the context of a single test execution (or executing a test partially while debugging) the environment uses less resources. When executing a test list with multiple tests in it there usage of resources is higher and thus the execution process might differ.
This is why we recommend using different timeouts and wait steps to keep the execution at the desired speed and not allowing it to outrun the tested application. Also test lists could be used for execution on multiple different environments which might perform differently. So before proceeding to test lists the tests should be stable enough.
The recommendations are to always use a verification for a state or text content of an element once there is any change in the page content. Another approach would be to include a wait for element step which will wait for an element on page before proceeding with the execution. That steps have a property 'EnsureStateIsCurrent' which (when checked) will verify if the state of the DOM tree is current.
If your website uses AJAX postbacks, you can set the AjaxTimeout value to something other than 0 (which is the default) on key action steps, such as clicking on a Submt or Next button which causes the UI to update. 0 means to not watch and wait for AJAX postbacks. A non-zero value, e.g. 30000, means wait that amount of time after performing the action for any outstanding AJAX postback communication with the web server to finish.
In certain occasions you might need to increase or decrease the timeout for a single element or adjust that globally for the project. Please find some additional information here.
I hope this will be useful for you!
Regards,
Elena Tsvetkova
Progress Telerik
Thank you for addressing this forum post.
One of the main tasks in the automation process is to ensure stable test execution. In the context of a single test execution (or executing a test partially while debugging) the environment uses less resources. When executing a test list with multiple tests in it there usage of resources is higher and thus the execution process might differ.
This is why we recommend using different timeouts and wait steps to keep the execution at the desired speed and not allowing it to outrun the tested application. Also test lists could be used for execution on multiple different environments which might perform differently. So before proceeding to test lists the tests should be stable enough.
The recommendations are to always use a verification for a state or text content of an element once there is any change in the page content. Another approach would be to include a wait for element step which will wait for an element on page before proceeding with the execution. That steps have a property 'EnsureStateIsCurrent' which (when checked) will verify if the state of the DOM tree is current.
If your website uses AJAX postbacks, you can set the AjaxTimeout value to something other than 0 (which is the default) on key action steps, such as clicking on a Submt or Next button which causes the UI to update. 0 means to not watch and wait for AJAX postbacks. A non-zero value, e.g. 30000, means wait that amount of time after performing the action for any outstanding AJAX postback communication with the web server to finish.
In certain occasions you might need to increase or decrease the timeout for a single element or adjust that globally for the project. Please find some additional information here.
I hope this will be useful for you!
Regards,
Elena Tsvetkova
Progress Telerik