We are using Test Studio in order to ensure the quality of our developed Silverlight applications. Therefore we execute 11 test lists (containing about 34 single test) every night automatically. Over the last few months, when the test suite grew larger, we identified some issues in the execution of our tests. It happens very often that some tests seem to fail at random although the tests application is working correctly.
The main reasons we identified when a test failed at random were:
- Element to be clicked was not visible/enabled yet because the screen wasn't ready. The test execution was simply too fast. --> "Unable to locate element" or the click hit the disabled element.
- Error presented by Test Studio: "Execute command failed"
- unknown reason
Hence we tried to make the tests more robust and stable against unpredictable cases such as longer application/element loading times, slow service requests, busy CPU,...
This is what we tried to stabilize the tests:
- inserted execution delays before critical steps
- inserted "wait for" steps (especially waiting for BusyIndicators)
- increased WaitForElementTimeouts up to 30s
- added general execution delay for the test lists between steps (100ms-500ms)
- kill browser process on close
But we do not want to insert manual execution delays before every step we record...
Unfortunately this did not solve the problem of randomly failing tests. We still have a failure rate of 15-30% for some tests (yes, we did a detailed analysis) just because the tests are not working correctly. This results in a very time intensive effort to maintenance the tests and analyze the results. And it is kind of annoying.
At this point the automatic execution of the test lists is done at least two times a day such that we can find randomly failed tests more easy.
So the question is: do you have any tips to improve the stability and robustness of the test further?
The main reasons we identified when a test failed at random were:
- Element to be clicked was not visible/enabled yet because the screen wasn't ready. The test execution was simply too fast. --> "Unable to locate element" or the click hit the disabled element.
- Error presented by Test Studio: "Execute command failed"
- unknown reason
Hence we tried to make the tests more robust and stable against unpredictable cases such as longer application/element loading times, slow service requests, busy CPU,...
This is what we tried to stabilize the tests:
- inserted execution delays before critical steps
- inserted "wait for" steps (especially waiting for BusyIndicators)
- increased WaitForElementTimeouts up to 30s
- added general execution delay for the test lists between steps (100ms-500ms)
- kill browser process on close
But we do not want to insert manual execution delays before every step we record...
Unfortunately this did not solve the problem of randomly failing tests. We still have a failure rate of 15-30% for some tests (yes, we did a detailed analysis) just because the tests are not working correctly. This results in a very time intensive effort to maintenance the tests and analyze the results. And it is kind of annoying.
At this point the automatic execution of the test lists is done at least two times a day such that we can find randomly failed tests more easy.
So the question is: do you have any tips to improve the stability and robustness of the test further?
9 Answers, 1 is accepted
0
Hello Aaron,
I am sorry to hear that you are experiencing these issues.
There could be different reasons why your tests fail randomly. The most common solution for these problems is to add a wait step, right before the failing step. You could try with using Wait for element's visibility is Visible or Wait for exists, and increasing the test step's Timeout property is needed. In this way Test Studio will wait up to the time set in the Timeout property, refreshing the DOM tree several times per second, and when the element is found will proceed with the next step.
You can also try the solution from this article.
If this is not your case, we will need more information about your test scenario.
1) Please send test failure logs for 3 of your tests (there should at least 3 files in each test’s export). You can export these to file from the test list result page if you drill down to each test.
2) In addition, please provide us with a copy of your project and if it is possible grant us access to your application so we can reproduce the issue on our end and give you a solution.
Looking forward to hearing from you.
Regards,
Velin Koychev
Telerik
I am sorry to hear that you are experiencing these issues.
You can also try the solution from this article.
If this is not your case, we will need more information about your test scenario.
1) Please send test failure logs for 3 of your tests (there should at least 3 files in each test’s export). You can export these to file from the test list result page if you drill down to each test.
2) In addition, please provide us with a copy of your project and if it is possible grant us access to your application so we can reproduce the issue on our end and give you a solution.
Looking forward to hearing from you.
Regards,
Velin Koychev
Telerik
0
Aaron
Top achievements
Rank 1
Rank 1
answered on 07 Oct 2013, 07:08 AM
Thanks for your response.
I will post some logs within the next days.
I will post some logs within the next days.
0
Hello Aaron,
Take your time and update this forum thread when you have more information.
Regards,
Velin Koychev
Telerik
Take your time and update this forum thread when you have more information.
Hope to hear from you soon.
Regards,
Velin Koychev
Telerik
0
Aaron
Top achievements
Rank 1
Rank 1
answered on 10 Oct 2013, 02:02 PM
Hello Velin Koychev,
Thanks for your patience. You find three attached files. The results.png shows a screenshot from the recent test results over a period of one week. As you can see some of the tests seem to fail randomly. The application to be tested works like charm, so every red result is a false negative. We need to put a high effort into maintaining the tests and analyzing the results. You will understand that it is stressful to sort out the false negative tests from the true negative test results. The repetition of the tests makes it more easy for us to filter these.
The three attached archives contain the logs of three randomly failed tests. We were not able to find the original fault of these test runs. Although we modified and tried to stabilize the tests again and again some of them still fail at random.
Unfortunately we are not able to give you access to the application to be tested or the complete Test Studio project.
Hope to hear from you soon.
Thanks for your patience. You find three attached files. The results.png shows a screenshot from the recent test results over a period of one week. As you can see some of the tests seem to fail randomly. The application to be tested works like charm, so every red result is a false negative. We need to put a high effort into maintaining the tests and analyzing the results. You will understand that it is stressful to sort out the false negative tests from the true negative test results. The repetition of the tests makes it more easy for us to filter these.
The three attached archives contain the logs of three randomly failed tests. We were not able to find the original fault of these test runs. Although we modified and tried to stabilize the tests again and again some of them still fail at random.
Unfortunately we are not able to give you access to the application to be tested or the complete Test Studio project.
Hope to hear from you soon.
0
Hi Aaron,
Thank you for attaching the log files.
Let's start resolving these failures one by one:
Pipe Broken -
Unfortunately there isn't a screenshot ImageOnFailure in the ZIP file, so I can't be 100% sure what was the reason for this problem. I can see from the log file the following line, right before the test failed:
LOG: Unexpected dialog encountered. Taking no action
UnableToFindElement -
From the screenshots I can see that the page was not fully loaded. So you need to add a wait step and increase the test step's Timeout property.
Regards,
Velin Koychev
Telerik
Thank you for attaching the log files.
Let's start resolving these failures one by one:
Pipe Broken -
Unfortunately there isn't a screenshot ImageOnFailure in the ZIP file, so I can't be 100% sure what was the reason for this problem. I can see from the log file the following line, right before the test failed:
LOG: Unexpected dialog encountered. Taking no action
One thing you can try is to set the UnexpectedDialogAction property to HandleAndContinue. This should handle all unexpected dialogs automatically when they show up. This will help you close all unexpected dialogs when you execute the tests as a part of a test list.
In order to set this property when executing from 'Quick Execution', you can override it in a coded step like this:
Manager.Settings.UnexpectedDialogAction = UnexpectedDialogAction.HandleAndContinue;UnableToFindElement -
From the screenshots I can see that the page was not fully loaded. So you need to add a wait step and increase the test step's Timeout property.
ExecuteCommandFailed -
You should add a wait step for the element that you want to select. This will allow the element to be fully loaded, before the select step is executed.
If you continue to experience problems, please record a short video to show us the issue you are experiencing. Jing is great for this.
I hope this helps.
Regards,
Velin Koychev
Telerik
0
Aaron
Top achievements
Rank 1
Rank 1
answered on 15 Oct 2013, 01:17 PM
Hello Velin,
thanks for your response. We will try to add more wait steps and execution delays.
One question to the ExecuteCommandFailed case:
Why do I have to manually add a wait step for the element to select? There is a WaitForElement timeout defined and WaitForElement = true with this step. I expect Test Studio to handle this automatically.
Another question to the UnableToFindElement case:
The log gives the error
The image captured on failure shows a fully loaded screen which has started to retrieve data ("Processing..."). If the screen would still be in a loading state the BusyIndicator displays the message "Screen is loading...".
The "missing" BusyIndicator should already be existent (inside the collapsed RadExpander) and I, again, expect the Test Studio to wait at least for the WaitForElement timeout of this step (which is 10000ms). But it doesn't.
We will look down on this issue further and inform you on the outcome.
thanks for your response. We will try to add more wait steps and execution delays.
One question to the ExecuteCommandFailed case:
Why do I have to manually add a wait step for the element to select? There is a WaitForElement timeout defined and WaitForElement = true with this step. I expect Test Studio to handle this automatically.
Another question to the UnableToFindElement case:
The log gives the error
System.ArgumentException: Unable to find 'radbusyindicator' element!The "missing" BusyIndicator should already be existent (inside the collapsed RadExpander) and I, again, expect the Test Studio to wait at least for the WaitForElement timeout of this step (which is 10000ms). But it doesn't.
We will look down on this issue further and inform you on the outcome.
0
Hello Aaron,
Thank you for the reply.
You are right that if there is WaitOnElements = true, the step should not fail immediately. It is hard to say what was the cause of this problem without reproducing it. Please provide us with a copy of your test and if it is possible grant us access to your application so we can reproduce the issue on our end and give you a solution.
The difference between the wait step and the WaitOnElements property is that the wait step will also refresh the DOM tree several times per second. Usually you don't need to refresh the DOM tree, but on some specific cases this might help.
Regards,
Velin Koychev
Telerik
Thank you for the reply.
You are right that if there is WaitOnElements = true, the step should not fail immediately. It is hard to say what was the cause of this problem without reproducing it. Please provide us with a copy of your test and if it is possible grant us access to your application so we can reproduce the issue on our end and give you a solution.
The difference between the wait step and the WaitOnElements property is that the wait step will also refresh the DOM tree several times per second. Usually you don't need to refresh the DOM tree, but on some specific cases this might help.
Looking forward to hearing from you.
Regards,
Velin Koychev
Telerik
0
Aaron
Top achievements
Rank 1
Rank 1
answered on 18 Oct 2013, 12:26 PM
Hello Velin,
Thanks for your reply.
Unfortunately it is not possible to grant you external access to the application.
We will try to add more wait steps before critical steps. Maybe the DOM tree refresh solves the issues.
Thanks for your reply.
Unfortunately it is not possible to grant you external access to the application.
We will try to add more wait steps before critical steps. Maybe the DOM tree refresh solves the issues.
0
Hi Aaron,
Thank you for the clarifications. I hope that these steps will help.
If you continue to experience these issues, please capture a Fiddler trace and attach it to this support ticket in a zip file. If you are unfamiliar with how to do so, this link will provide you with step-by-step instructions for download and use. Please make sure to enable 'Decrypt HTTPS traffic' and 'Store binaries' options (see attached image) before starting capture.
Velin Koychev
Telerik
Thank you for the clarifications. I hope that these steps will help.
If you continue to experience these issues, please capture a Fiddler trace and attach it to this support ticket in a zip file. If you are unfamiliar with how to do so, this link will provide you with step-by-step instructions for download and use. Please make sure to enable 'Decrypt HTTPS traffic' and 'Store binaries' options (see attached image) before starting capture.
Hope to hear from you soon.
Regards,Velin Koychev
Telerik