Test Studio as part of VSTS Release

Hi Zac,

Thanks for all shared notes on the setup you would like to cover. 

If you have created Test Studio test files (*.tstest), there are two options you can go for to implement the test execution in a VSTS (Azure DevOps) build. One will be to use the Test Studio runner called ArtOfTest.Runner.exe or the VS runner VSTestConsole.exe. Please find some additional notes about the two possible approaches below: 

Integrate the ArtOfTest.Runner.exe in VSTS Build 
1. Here is the sample article with steps you can follow to include a task in the build to run tests with the Test Studio command line runner. 
2. Using this option
- the project gets compiled automatically compiled before starting the test execution. 
- results are automatically stored in the local project copy results folder and can be also exported to an external location

Execute tests with VSTestConsole.exe 
1. The VS test runner requires some additional adjustments and parameters passed in order to execute the Test Studio tests - these are described here. Please note the listed file paths are for Visual Studio 2015 default installation and will probably differ from the installation of VS 2017 you mentioned. 
2. The test execution status and possible failure details are outputted in the console and no results are generated in the Test Studio project. 

Basically, these are the specifics you can keep in mind when taking the decision which option will suit you better. In case there are any other questions or concerns you might have, please do not hesitate to get back to me. 

Thank you. 

Regards,
Elena Tsvetkova
Progress Telerik

Thanks Elena,

We are probably going down the root of VSTestConsole.exe. We cannot find a way to create an aiilist when working in Visual Studio for the ArtOfTest solution to work.

VSTestConsole will be ran as a release task 'VsTest' in VSTS pipeline and on a dedicated machine. Regards to licensing, how does this work do we need a full blown licence for Test Studio on the test machine that has Visual Studio installed as you point out we need a TestAdapter? We already have a licence for Telerik Runtime Addon but dont see how this can work with VSTestConsole.

 

 

Hi Zac,

Thank you for getting back to me and I am glad to know my previous comments helped you to decide which option will suit you more. 

As you have pointed, it seems that the TestAdapter is not being deployed with the Run-time edition and this will be further investigated and fixed on our end - here is a bug report, which you can follow. 

Please, give a try to the workaround and get back to me in case still there any troubles with that. 

Thank you for your understanding in advance. 

Regards,
Elena Tsvetkova
Progress Telerik

VSTest console is working well with Test Studio is there a way to utilise multi browser testing with this scenario?

Hi Mike,

Thank you for contacting us with this query. 

Since you are working in a Visual Studio solution, you probably are familiar with the settings you can switch for the test execution from the Visual Studio Test Explorer. These settings can be also used for the test execution with the VSTest.Console.exe, passing the file to the /Settings option of the VS command line runner. Though, it is important to note that the Multibrowser Execution setting from the Test Studio test list settings is not working in the context of Visual Studio project and cannot be set for the VS Test Explorer or the VSTest.Console test runs. 

However, still there is a solution to cover the requirement for running a test against the different browsers - prepare few separate settings files defining the desired browser and possibly other settings. Then, add separate execution tasks passing the different settings file to simulate multibrowser test runs.

In addition to the above suggestion, I would like to mention that you can pass multiple test files in a VSTest.Console command separating these with interval - that way all these will be executed from this task. Below is an example command: 


I hope these details are helpful for you. Of course, in case of further questions, please let me know. 

Regards,
Elena Tsvetkova
Progress Telerik