I have multiple tests that all pass successfully when run on Tests and Test Lists tabs within the test studio application, but when set to run at a specific time on the scheduler, they fail at some point in the middle of the test.
Conditions are otherwise the same (I have a windows process set up to make sure that they're even connected to the same router).
Here's an example log from one of the failed tests. These same failed steps pass when I run the test myself without the scheduler. I'm stumped. Anyone run into this or have suggestions on fixing it?
Lance
Overall Result: Fail------------------------------------------------------------
'9/19/2016 8:10:20 AM' - Executing test: 'Submit Contact Form', path: 'Web\Forms\Contact Form\Submit Contact Form.tstest.'
'9/19/2016 8:10:20 AM' - Using .Net Runtime version: '4.0.30319.42000' for test execution. Build version is '2015.2.723.0'.
'9/19/2016 8:10:20 AM' - Starting execution....
'9/19/2016 8:10:21 AM' - Detected custom code in test. Locating test assembly: Spire Web.dll.
'9/19/2016 8:10:21 AM' - Assembly Found: C:\Users\Asus\Documents\spiredigital-telerik\SpireDigital\bin\Spire Web.dll
'9/19/2016 8:10:21 AM' - Loading code class: 'Spire_Web.Submit_Contact_Form'.
------------------------------------------------------------
------------------------------------------------------------
'9/19/2016 8:10:21 AM' - Using 'InternetExplorer' version '11.0' as default browser.
'9/19/2016 8:10:21 AM' - Using 'http://spiredigital.com' as base url.
'9/19/2016 8:10:21 AM' - 'Pass' : 1. 'Maximize' the browser.
'9/19/2016 8:10:23 AM' - 'Pass' : 2. Clear Browser Cache
'9/19/2016 8:10:24 AM' - 'Pass' : 3. Navigate to : '/'
'9/19/2016 8:10:25 AM' - 'Pass' : 4. Wait for '500' msec.
'9/19/2016 8:10:25 AM' - 'Pass' : 5. Enter text 'Automated Test Email' in 'NameText'
'9/19/2016 8:10:25 AM' - 'Pass' : 6. Wait for '250' msec.
'9/19/2016 8:10:26 AM' - 'Pass' : 7. [Contact_Form_Submission_CodedStep] : Enter text '' in 'EmailEmail'
'9/19/2016 8:10:26 AM' - 'Pass' : 8. Wait for '250' msec.
'9/19/2016 8:10:26 AM' - 'Pass' : 9. Enter text '3031234567' in 'PhoneText'
'9/19/2016 8:10:27 AM' - 'Pass' : 10. Wait for '250' msec.
'9/19/2016 8:10:27 AM' - 'Pass' : 11. Enter text 'Automated test, please disregard.' in 'MessageTextArea'
'9/19/2016 8:10:27 AM' - 'Pass' : 12. Wait for '500' msec.
'9/19/2016 8:10:27 AM' - 'Pass' : 13. Desktop command: LeftClick on SubmitButtonTag
'9/19/2016 8:10:28 AM' - 'Pass' : 14. Wait for '500' msec.
'9/19/2016 8:10:59 AM' - 'Fail' : 15. Wait for element 'ThankYouH2Tag' 'is' visible.
------------------------------------------------------------
Failure Information:
~~~~~~~~~~~~~~~
Element is visible verification failed! Expected: True, actual: False.
------------------------------------------------------------
'9/19/2016 8:10:59 AM' - Detected a failure. Step is marked 'ContinueOnFailure=False' aborting test execution.
------------------------------------------------------------
'9/19/2016 8:10:59 AM' - Overall Result: Fail
'9/19/2016 8:10:59 AM' - Duration: [0 min: 37 sec: 97 msec]
------------------------------------------------------------
'9/19/2016 8:10:59 AM' - Test completed!
3 Answers, 1 is accepted
0
Hi Lance,
The error logged in the execution log provided is due to an element that is not yet present during the execution. You should consider that running tests on a separate machine depends on its performance and network speed and the application performance speed may be different from your local machine. Though I am not sure you schedule your tests on a different machine. Other dependency may be that the application does not respond in the same way when scripts are performed at the time when scheduled.
Anyways, my recommendation is to enlarge the execution delays in this case to ensure the page will be loaded and the element will be available. Let me know if further assistance will be required.
Regards,
Elena Tsvetkova
Telerik by Progress
The error logged in the execution log provided is due to an element that is not yet present during the execution. You should consider that running tests on a separate machine depends on its performance and network speed and the application performance speed may be different from your local machine. Though I am not sure you schedule your tests on a different machine. Other dependency may be that the application does not respond in the same way when scripts are performed at the time when scheduled.
Anyways, my recommendation is to enlarge the execution delays in this case to ensure the page will be loaded and the element will be available. Let me know if further assistance will be required.
Regards,
Elena Tsvetkova
Telerik by Progress
0
Lance
Top achievements
Rank 1
Rank 1
answered on 22 Sep 2016, 03:13 PM
Hi Elena,
Thank you for responding.
The tests are being run locally on the same machine where Telerik Test Studio is installed. I've experimented with extending explicit wait times before, and found it had no impact on the reliability of the tests. Last night, I added explicit wait times of a minute between steps and the application still failed at the same point in the scripts.
I've run these tests manually (may times) at the same time of the morning and I've never had any issues with the tests successfully executing.
0
Hello Lance,
I am sorry to hear that you still are facing the same behavior. Though this is specific for your application and environment having in mind that the example error is related to the app UI and a missing element. Please note that manual execution is quite different than the automated test run and it is expected that manual runs will be successful.
Since you say the tests are failing at the same time I would like to ask you to observe the remote execution and see the exact state of the tested application - is the element the test waits for available on the page at all? What happens if you change the verification to check if element exists? Please note that the web page may be differently rendered on the other machine and this could cause the described behavior. Please give a try to the modified verification and let me know what the results would be.
Thank you in advance for the cooperation! I hope to hear from you soon.
Regards,
Elena Tsvetkova
Telerik by Progress
I am sorry to hear that you still are facing the same behavior. Though this is specific for your application and environment having in mind that the example error is related to the app UI and a missing element. Please note that manual execution is quite different than the automated test run and it is expected that manual runs will be successful.
Since you say the tests are failing at the same time I would like to ask you to observe the remote execution and see the exact state of the tested application - is the element the test waits for available on the page at all? What happens if you change the verification to check if element exists? Please note that the web page may be differently rendered on the other machine and this could cause the described behavior. Please give a try to the modified verification and let me know what the results would be.
Thank you in advance for the cooperation! I hope to hear from you soon.
Regards,
Elena Tsvetkova
Telerik by Progress