Tests shown as "Not Run" with remote execution

Hi Aleksandar,

Thank you for the detailed description of the setup you have prepared on your end. 

To be sure I am familiar with all the details I will list few assumptions below and please, correct me in any of these if necessary: 
1. There is a local machine with Test Studio standalone installed, on which you develop the test project 
2. There is a VM (the Win 7 machine) with Test Studio Run-time installed. 
3. The VM machine hosts the Scheduling and Storage services. 
4. The Test Runner (or Execution client) is connected to the configured Scheduling service. 
5. The project on the local machine is also connected to the Scheduling service. 
6. Optional you can connect the Execution client on your local machine to the same Scheduler. 
7. The Execution status view shows the current status of services, storage database and the connected execution clients. 

If all of these are correct, you should have any results stored in the Calendar view from the attempts you have performed to run tests on the VM machine. Can you please export any of these and send a sample for further details? You can use the emailing option to export the aiiresult file, which is the Test Studio default result file. 

In addition to that, I will appreciate if you generate the complete application log files from both the local and VM machines - you can access both these in the Execution status view by double clicking the two machines. To capture a valid log file, please enable/clear the logs for both machines, then trigger a test list run on the VM machine and once this fails, collect the log files from the two machines and generated result in calendar. These can be then sent zipped via this thread. 

One more important note is that the VM execution machine requires an active user session to execute any tests - is this prerequisite also met? 

Thanks in advance for your cooperation. I am looking forward to hearing back from you. 

Regards,
Elena Tsvetkova
Progress Telerik

Hi Elena,

Thanks for the quick response. Both machines (standalone and execution only) are VM's with Windows 7. Both have scheduling and storage services installed and running. 

We didn't connect the machines to the same scheduling server. We've corrected that, but now we have a bunch of errors in logs. The end result is still the same, tests are shown as "not run".

Execution machine is VDE-EPIPC04, Test Studio standalone machine is VDE-EPIPC10 in our case. If we set both machines to have scheduling URL to #10 no machines are displayed in the remote execution list. The only way to see the machines is to set Execution clients to have scheduling URL on #04 (the execution machine). Again, both machines have storage and scheduling server installed and running.

I have attached the logs and screenshots for both machines to this message.

Thanks,

Aleksandar

Hi Aleksandar,

Thanks for the additional notes for the environmental specifics on your end. 

First thing I want to point to is that the Test Studio Scheduling setup requires a single Scheduler and Storage services host. It acts as a centralized controller for the execution machines - these, on the other hand, can be multiple machines and the minimum requirement is to have the Run-time installed. 

Then, next thing to have in mind, is that the Test Studio standalone default installation does not include the services, where the Run-time default installation adds these automatically. Probably this is the reason you have these installed on both machines. However, this is not supposed to cause any troubles, if you use the services on one of the machines. 

So, having said the above, here is how you can proceed: 
1. Choose which of the machines will be used as a Scheduler and Storage services host. You mentioned that the remote machine's (VDE-EPIPC04) Scheduler seems to be configured and working, when you connect the Execution clients to it and I you can use it. 
2. If this is the decision, please, open the Scheduling configuration wizard on that remote machine, and click on Apply for each of the tabs. You can start it by typing 'Configure as Scheduling Server' in the Windows start menu. 
3. Please collect screenshots of each tab, after clicking the Apply button. 
4. Expected is that all services are started successfully. 
5. Open the Execution clients on both machines - VDE-EPIPC04 and VDE-EPIPC10, and connect these to the configured Scheduler on VDE-EPIPC04. 
6. In the Test Studio standalone on machine VDE-EPIPC10, connect the project to that same Scheduler on VDE-EPIPC04
7. The Execution status view main window should now display the two machines as Execution servers, and information for the configured Scheduler and Storage. Please, prepare a screenshot of the main screen in this view. 

These steps are only to confirm if the setup is correctly configured, as I suspect you managed to get this working when reading the log files. So, in case you have performed the above, you don't need to reconfigure the setup again. 

Next thing I would like to ask for is a simple test to verify if there are any further obstacles for the setup: 
1. Add a new empty web test in the project and record a small sample test - simple navigate step and a click, or anything else. 
2. Add this one to a test list and run this one remotely on the other machine. 
If that test list is running successfully, we can be sure the Scheduling communication between the machines is not interrupted or restricted and can proceed further. 

The log file you managed to generate and send me points that there is a successful attempt to run a test on the remote machine, but there are compilation errors for missing elements. These details will be also listed in the failed results displayed in the Calendar - to see the complete log for selected result, use the Execution Log button (please, see the screenshot attached). Please, find additional details on these errors and how to solve them in this article. 

Once these missing elements errors are resolved, you should be able to run the tests on the remote machine. 

I hope these notes will be helpful for you. Though, in case of further questions, please, get back to me. 

Regards,
Elena Tsvetkova
Progress Telerik

Hi Elena,

looks like it's compilation errors. In coded steps we have used "pages" reference. This causes the errors.

I've seen that there is an option "get latest version of tests from TFS". We are using VSTS (Azure DevOps) and its built in GIT repository. Is there a way to automatically get the tests from VSTS GIT in any way? Would this be resolved if we had the project physically present on the execution machine or do we have to completely refactor the coded steps?

Thanks,

Aleksandar

Hello Aleksandar,

Thank you for taking time to review the shared resources. 

Let me please, add few more details for the compilation issue you encountered - as you probably have read, when a test list is being executed remotely or scheduled, only the related project files are being deployed to the remote machine. So, if there is a test, which refers to an element from the repository, which is not referenced to that same test, and is also not referenced in any of the rest tests in the test list, the remote execution will end up with the compilation error in question. 

However, if we use the example from the article and include the two tests to a test list - both the RunTest and Test1, the test list will be successfully executed on a remote machine. The reason behind this is that the Pages file is being dynamically generate in run-time - some further details for how the Pages file is used can be found here. 

On the other hand, if using the option 'Get Latest from TFS', the whole project will be deployed to the remote machine and thus, there will be no missing elements. And as you mentioned you are using Azure DevOps as source control, it would be the easiest fix for the errors, but the option is applicable for the default TFVC repos only - the one you use, Git repo, will not even enable the 'Get Latest from TFS' option.

The next available approach will be the listed in the article - to include find expression statements, instead of referring the element from the Pages file. However, this may cause you troubles in the future if you need to modify the find expression for the element - each test step will need to be updated. 

Still, there is a suggestion, which has not being discussed so far but I think it is the most appropriate one. This will be to add the element to the test, or in other words to add a reference in the test to that same element. Following the steps below will ease you in adding the element: 
1. Open one of the tests claiming that it has a missing element. 
2. Execute the test using Run-To Here from the Step context menu - you can select a step which is supposed to be on the same page, where is the missing element.
- If this will not be an option, you can execute the test to any step, pause the recorder, and manually navigate or interact with the page to get to the desired page. 
3. Then, once on the page, which contains that element in question, highlight it and choose to add it to repository. 
4. Refresh the elements in the Elements explorer and save the changes. 
5. Executing the test remotely should now be successful (if all missing elements are added to the test). 
6. Some additional notes on this solution can be found here. 

The reason to recommend this solution is that the maintenance of elements will be easier, as you will need to modify an element once, and then save it for all tests, where it is used. 

I hope these additional notes will be also helpful for you. And, of course, if there are any further questions, please do not hesitate to get back to me. 

Regards,
Elena Tsvetkova
Progress Telerik

Hi Elena,

sorry for the late response. We have been experimenting a little bit.

Changing all coded steps and removing all "Pages" references helped. Our tests are now running on remote machine.

Thanks for your help,

Aleksandar

 

P.S. we have noticed something. If we tried to run the list and execution failed due to "Pages" reference and we remove the trouble causing test from the list when we tried to execute the list again we were not able to. Because it looks like TS keeps the older build or references somehow. We had to refactor the said test, make a successful run including that test and then remove it. Only after that we were able to execute the list without that test and without errors regarding that test.

Hello Aleksandar,

Thanks for getting back to me - I am pleased to know you managed to sort out and fix the errors. 

Here are some additional notes In regards the failing remote runs even after you had removed the trouble test.

In the Test Studio Scheduling setup the Telerik Storage service uses MongoDB as a storage database - this is the place where the project files are stored and deployed to the execution machines (in the case when 'Get latest from TFS' option is not used). Test Studio allows you to update these manually or take advantage of the automatic upload of changed files. The dialog is being triggered whenever you save the whole project - using the Save All button in the upper left corner of Test Studio window or by pressing Ctrl+Shift+S on your keyboard.

What you should be aware of for the automatic upload is the following - if you select an option what to do with the changed files and select to not show that dialog again, the option you have selected is being triggered for each Save All action. If you are unsure what option you have selected and the dialog is being removed you could bring it back as follows:
1. Open the project's root folder and open the Settings.aiis file with a text editor (Notepad++ for example).
2. Double check the Project ID value and note it somewhere.
3. Go to C:\Users\<yourUserName>\AppData\Roaming\ArtOfTest folder and locate a file named PerUserSettings.json.
4. Open that file with a text editor again and locate the Project ID under "SkipUploadScheduledJobToStoragePromptPerProject" setting. Delete the project's id, if listed there and save the file.

Then when you start the project again, make any change and select Save All, the dialog will prompt you again and you could choose the desired option to automatically update the files. That way all changes will be reflected in the database as well whenever you Save All.

I hope this will be also helpful for you in managing the Test Studio project. Of course, if there is anything else I could be helpful with, please do not hesitate to get back to me again. 

Regards,
Elena Tsvetkova
Progress Telerik