Scheduled Test List Not Running the Latest

*using version 2014.3.903.0

Hi Sean,

There are a couple of possibilities for the symptoms you describe. Let me start with a description how the scheduling server system works.

Tests and test lists are stored in our Storage service, which uses a SQL DB for the actual physical storage. Nothing ever gets deleted out of this DB. We admit this is a flaw and are actively working on a new design to overcome this flaw. Thus deleted tests and test lists will still show up on the remote executor when you run any test list.

When you schedule a test list to execute, Test Studio will try to intelligently push into storage only those tests that are part of that test list and have changed. It does this by comparing the files date & time stamp to the one stored in the SQL DB for the test. If what's on disc is newer than what's in storage it will get pushed, else it will be skipped.

Where this may break down is if you call a subtest via a coded step. Test Studio is not yet intelligent enough to scan coded steps for dependent subtests. As a result that particular sub test won't get pushed into storage and the latest version won't get used during test list execution.

Another possible hazard is if tests were copied outside of Test Studio, such as in Windows explorer, the ID of the tests and test lists, which is stored within the file itself, will become duplicated. Now when you schedule this copied test or test list, it will conflict with what's already in storage.

If any of the above hazards sound like they apply to you, let me know and we'll work on overcoming the problem.


 Regards,
Cody
Telerik

Yes I use Test as a Step for many of my tests because I have a lot of repetitive and reusable test cases, These are also duplicated for environmental purpose for example I have a Stage version and a Production version for logins because they vary between environments. My hope was I could just enable and disable these Steps and the scheduler would pull this latest change. This duplication is not limited to test as a step either Nav URLs get the same setup for environments and enabling/disabling these single steps is not always picked up either.

I did run into the issue of copying using windows explorer and had the issue of it running tests lists from another project so now I use the Add Existing Function to create a new project and reuse existing test.   

To get around all this my solution has been to create a new project, add existing test 1 by 1, delete any of the environment specific test steps then create a new test list and run that.  This adds new issues as TestStudio will sometimes crash.

Hi Sean,

Yes I use Test as a Step for many of my tests because I have a lot of repetitive and reusable test cases

Excellent! That's exactly what we designed it to be used for.

I did run into the issue of copying using windows explorer and had the issue of it running tests lists from another project so now I use the Add Existing Function to create a new project and reuse existing test.

Glad to hear that. That is the right way to do it. Add Existing Test will replace the tests GUID contained in the original .tstest file with a newly generated one, guaranteed to be unique. Test steps do not have GUID's. You can duplicate those, enable/disable them all you want and this will not cause any problems.

My hope was I could just enable and disable these Steps and the scheduler would pull this latest change

Yes it should work that way provided:
1) You save the changed tests to disc
2) The changed tests are included in the test list (not only referenced by a coded step).

Given the above 2 conditions, Test Studio should be picking up the changes, pushing the updated tests to our Storage service, which will cause them to be used during execution of the test list run. If you think you have met both of these conditions and it's still not working as expected, do you think you can demonstrate the problem to me in an online meeting?

Regards,
Cody
Telerik

I'm facing a similar issue. I first updated a test to included a coded step, and then I ended up converting the coded step to a test as a step, but the latest version is not running when I execute the test locally as part of an established test list. I even tweaked and/or added a few comments to the test, but the list is still executing the old version of the test.

It would be nice if the "Upload latest project files" option on the results tab would give you a count of the steps that were uploaded or some type of status message. Right now, you have no way of knowing whether the upload was successful, or if it even did anything.

 Hi Cody

I am experiencing very seriouse issue with our Automatic test setup which consist of 8 test agents (TS Runtime) and one controller (TS Ultimate) with scheduling . The storage is on a SQL Server and the test cases (the project) is version controlled by TFS.

When executing test list DISTRIBUTED then all the 38 test cases in the list fails. If I target the test list to be excuted on only one Test Agent  then the test cases will pass.

 This is a show stopper for us and we need an urgent assistance.

The controller is on windows 2008 R2 server and the test agents are mix of Win7, Win8 and win8.1.

I am using version 2015.2.723  on all machines and and selecting Get latest from TFS.

I appreciate your help and many thanks in advacne.

Br.

Ameer

 

 

 

 

 

 

 

 

Hello Ameer,

Let's keep the communication in the ticket you have opened for this behavior without updating other threads that are not related to the issue you are experiencing.

Thank you for your understanding.

Regards,
Ivaylo
Telerik