distributedlife

I needed an automated way to run our service tests each night with the executed tests being selected from the current cycle. We couldn’t tie this into a continuous integration server because of the complexities of trying to deploy 40 interlinked services based on service change as well as database structural changes. The latter of which did not have CI

To get around these issues I wrote a small console application that is scheduled to run at 2 in the morning (it has be done after midnight as your Quality Centre connection is reset at midnight).

This application runs until 8 in the morning when I terminate it regardless of how far it has gone. There are other jobs to run (I’ll describe these another day) that are dependent on the results of these tests.

I can’t provide source yet because it dependent on how we organise our test projects. We have service tests we don’t want to run any more. These are grouped into release folders and we only run the service tests for the most recent release. If you follow my recommendation for service test structure within QC then you could use this tool without modification.

The app has the following tasks:

• Setup release/cycle structure and test lab
• Execute tests that have never been run in the current cycle
• Execute tests that have not passed and have not been run today.

The last two are very important. We have over 10,000 service tests and these take more than 6 hours to run at the moment. This means it has to run over more than one night. If we are not careful about the order we run then we will run the same tests every night.

So it’s always best to run tests that have not been run before first. Then run all the tests that have failed in the cycle. The test may have been fixed during the day by the service tester or it may continue to fail. We ignore failed tests that have been run today because these tests were ‘tests that have not been run’ but are now classified as ‘failed’ tests in the second run. We already know the outcome so we skip them.

Setup release/cycle structure and test lab

For reporting purposes I require our service tests to belong to a QC release. So there is a configuration file that specifies the release. If this release is not found then we create it.

Within this release must be an open cycle. If there isn’t one we create one; otherwise we get the newest cycle that meets our requirements.

In the test lab we look for a folder named after our release. We then find the folder for our cycle. If one is not found then we find it and link it back to the cycle. Under this folder we create a folder for each service.

Under each service we create a test run for each endpoint and add all the tests for that endpoint into the test run. We also create a test run for service level tests.

These test runs are then execute as per the process outlined above.

Benefits

The benefits of this setup is that when a cycle is closed (for whatever reason) a new one is created and a new test lab is created and all the services are assigned to it for retesting. We can come up with a job that manages our cycles and automatically from that comes the automated test runner where each cycle commences with a re-execution of all tests.

All I used for this project was C# and the Quality Centre OTA API.