Command Line - Attaching results to email

Hello Matthew,

Test Studio provides two command line runners, which provides different options, depending on the execution mechanism they actually perform. To be more precise in the details, I will list below their specifics in separate. 

The ArtOfTest.Runner.exe

The AOT.Runner.exe is the official command line runner of Test Studio, which provides plenty of options to customize the different runs - you can use settings file to adjust the test execution. or output the generated results to a separate folder, or generate different type result files. The type of execution this runner performs is local execution, e.g. running a test list through the Run Locally button. Thus it does not require the installation of the Test Studio Services. 

The Telerik.TestStudio.CommandLineClient

The mentioned command line client is using the Scheduling setup, so the options it provides are different - they define the necessary adjustments related to the scheduling service. These, however, are not quite thorough and are not covering any possible needs you may encounter. There are few feature requests related to improving this command line client options and you can vote for these to increase their customer importance. You can find them on the following links:
- Add the ability to specify what attachment to be sent 
- Specify the execution machines to execute the test list on 

I hope these details will be helpful for you. However, I remain available to provide further best practices or ideas how you can accomplish the requirements for the project you work on. Having that said, please, do not hesitate to get back to me with further details for the scenario you need to cover. 

Thank you in advance for your cooperation.

Regards,
Elena
Progress Telerik

Elena,

Our goal is the following:

• Trigger test list(s) on our Telerik Test Studio Runtime server as part of our deployment process. We use Octopus, which can fire off PowerShell scripts to trigger command line commands, etc..
  • Note: we want to trigger the tests in our QA's domain - this is different than where the build occurs and the TFS server lives.
• Send the results, with as much detail (test lists files, etc.), to our team. Our team consists of 6 people. Essentially, we want to know when the Test Studio suite has finished running and an easy way to access the results.

Additional Note: The user firing off the PowerShell command (and thus the tests) will be a service account - not the Telerik Runtime account.

The CommandLineClient doesn't provide any attachments, which is a downfall. What is the best way for the QA user to drill down into the results when the CommandLineClient is used?

The ArtOfTestRunner doesn't appear to provide emails at all. How can we inform the team that the tests have finished/failed, and what the results are? Is the fact that it is a local execution a problem since we are using Telerik Runtime (and thus scheduling and execution servers)?

Hi Matthew,

Thank you for sharing detailed description of the requirements you need to cover - these are really helpful for me to provide you further details on what options you have. 

The first important note to add is in regards the Test Studio tests execution in continuous integration environment - two of the requirements to build such a setup is to run the build agent in console mode and to ensure an active logged user session on the machine on which the tests will be executed. Please, see additional notes on the topic in this article. In general, the Test Studio tests are functional tests and require user session - depending on what actions are executed throughout the tests, these may also require unlocked desktop. You can read further details on that here. 

So, having said the above, I guess you will need to reconfigure the Octopus build environment as per the recommendations. 

As a next topic in our discussion I will focus in the needs you listed and will again list my comments separated for the two CLI runners - know when the Test Studio suite has finished running and an easy way to access the results.

The ArtOfTest.Runner.exe 

The options for AOT.Runner test run, which can be useful for your needs, are ("out") to output the results into an alternative folder and, possibly, to automatically generate an exported version of the results to html ("html"), for example. This output folder can also be a shared drive folder, which everyone in the team can access, including the user logged on the agent machine. Outputting the results in any other than the default format, will allow anyone to review these, even if there is no Test Studio installation on the machine, from which the shared result folder is reviewed. 

I am not familiar in details with the features of Octopus, though, I suspect that there should be some post build tasks which can cover the notification when the test list run is finished. Have you explored such an option? 

Test Studio allows you to extend your test runs with custom code execution before or after each test run. To implement such custom solution, you can build an Execution Extension and include sending some notification in the OnAfterTestCompleted() or OnAfterTestListCompleted() methods - these are being triggered by each end of test or test list, respectively. 

The Telerik.TestStudio.CommandLineClient

This CLI runner allows you to send an email - however, its current format cannot be modified and it can be considered as a note that the run is finished. The generated results will be stored into the Storage database (which uses MongoDB as a provider). The options to review these is within the deployed Test Studio project, or from the Executive Dashboard hosted on that build agent machine - this can be accessed from anyone in the team using the same domain. 

As a side note, I want to sort out what I have in mind when saying local and scheduled execution in the terms of Test Studio. Actually, the difference is whether the tests are triggered to be executed through the Test Studio Scheduling and Storage services, or not. The main difference for these runs is where the results are stored - local run (using AOT.Runner or Run Locally in the project Test Lists tab) outputs the results into the project's root folder under Results sub-folder, and remote/scheduled run (using the Telerik.TestStudio.CommandLineClient or the Run Remotely button in the Test Lists tab) the generated results are stored in the Storage database. 

I also took the time to have a look on the Octopus features and capabilities and also noticed that it can be integrated with Jenkins, for example, in order to trigger the tests run. Test Studio can also be easily integrated in Jenkins where the results can be added for review as well. Based on the Octopus listed suggestion Test Studio also integrates with TeamCity, Bamboo, Azure DevOps and others. 

I hope that these additional notes will be of further help in our discussion. Please, do not hesitate to elaborate further about the Octopus setup you have - since I am not familiar with this tool in thorough details, I may have misread any of the suggested options to cover the final requirements. In addition, Test Studio offers plenty of different approaches for execution and results reviewing to choose among and based on the particular needs, you can best decide which will suit at most. Still, I remain available to continue the discussion if you have any doubts or concerns. 

Thank you once again for your cooperation.

Regards,
Elena
Progress Telerik