distributedlife

As a product itself HP Service Test doesn’t have the best support for different test configurations. As a copy-paste job of LoadRunner it’s in the mindset that each “Test”, which I will call a “project”, is a single executable test case. This makes sense for LoadRunner but as anyone who has had to do service testing will know; there are lots of test cases per service and the maintenance and management of the Service Test project files is a nightmare.

There are several factors that need to be considered when you decide how you structure your tests and projects; how much time you can allocate to configuration, how many your environment changes you are likely to experience, the number of testers you have working on each service, what kinds of metrics are required and how detailed they need to be, whether you have Quality Centre and even the number of tests per service.

I’m going to look at the options that are available and which one is I recommend. The important thing to note is that all of these configurations have significant downsides.

The HP Way

The default configuration from HP is that each project is one test case. If you own a licence to Quality Centre then this makes a lot of sense. Create a test case in QC and you launch Service Test from there. Write the code and you’re done.

This integrates with the test execution features of Service Test and automated behaviour when the test runs fail.

If you make use of the services tab you can generate a plethora of graphs that would have upper management ecstatically swimming through their own drool.

The HP Way gives you more flexibility in naming your test cases. This is because you will use a QC Test Case object for each test. This will give you around 270 characters per test case with several special characters included. All other scenarios use actions as test case and these are each limited to 64 characters and no underscores or special characters.

The downside to the HP Way is that it becomes increasingly difficult to maintain these projects. For some reason there is no useful link between a service end point that is added to the services tab and the one that is defined in the project file.

If your change service location, deploy to a new server for example, then you will have to update each project file. If you have setup the project files the way HP suggests then you have a painful find and replace job to undertake.

If you happen to support side by side deployment then you may have multiple instances of the same service deployed on the same machine. To switch between the two you’ll have to do the same configuration management process.

Multiple these by the number of tests per operation (I average about 80), the number of operations on a service interface multiplied by the number of services in your project and you start to have thousands of tests. That equates to thousands of Service Test project files that you have to manually update.

Pass.

I wrote some Perl to change the end point URL automatically but it’s not a great solution. A last option when you’ve got lots of projects to fix infrequently. If you use QC and you want to run the script then it has to be run against the QC server file system to which access should not handed out like candy. This then adds a burden onto the person who does have access, to run a script each time an environment change occurs.

If you do heavy parameter substitution or make use of helper methods to reduce the complexity of tests you write, the QC Services tab coverage does not get updated. I can’t seem to ever get the QC Services tab to show operation coverage but if it’s anything like the operation coverage in Service Test itself then it won’t be any good. The parser they use to determine operation coverage doesn’t handle parameterisation or helper methods. I’m not going to start writing bad code just to support metrics and neither should you.

A Project per Operation

This configuration is what I recommend if you are the one doing the work. For each operation on a service interface you create one project file. Within that file each action is one test case.

The configuration problem I talked about earlier goes away. A service with eight operations has eight test projects and it only takes about 20 seconds to change a service end point for each project.

One benefit of having each test represented as an action gives you an easy way to refer to existing tests (action) that have been written; identify common code; and replace it with a suitably abstracted method. We’re talking Vanilla C here so there isn’t abstraction that can be done much you can aside from writing a common subroutine.

When the project is run you can call done on each operation with confidence. Your project will probably take a few minutes to run in total which is manageable.

The only issue with this of course is that calling done can take a long time. If you have a bug that won’t be fixed in this release, or ever, you have a project that will be reported by QC as failing. It doesn’t look good on reports and it makes it difficult to quickly run the tests to ensure nothing has broken. All you get is a fail and you have to interrogate each failure to make sure that is supposed to fail.

You’re then going to have to manage the pass, fail and defect states manually in something like a spreadsheet. This isn’t ideal if you’ve spent a lot of money on QC.

Which brings me to the negative side of this configuration; metrics go out the window. If you have QC and you’ve linked everything together. You will have only eight “tests” to measure metrics against and if any test within your project fails then the entire project is recorded as a failure in QC.

If you want metrics from your service testing efforts then you will have to do it externally from QC and Service Test. It won’t be automated, which isn’t a fabulous result.

A Project per Operation Aspect

This is how I configure my service tests now. I started originally with “A Project per Operation” but now this is how I work. This is a setup like “A Project per Operation” but to compensate for the lack of metrics being generated we split the projects up into aspects.

Some examples of aspects are Security, Exception Management, Auditing, Input Validation, Deployment and compliance. Large chunks of functionality testing can be broken into discrete aspects as well. For instance I have just over 50 tests that relate to “title validation” on each service operation. I would consider “title validation” as an aspect. Pull those tests into a separate project file.

This increases the granularity of the test cases and therefore more accurate metrics while not increasing the project count so much that you have to allocate a day to configuration management.

At the moment I would consider 6 to 10 as the number of aspects that is manageable. Any more than this and you will start to have configuration management issues again.

A Project for Per Service

This one isn’t worth discussing in any real depth. Having one project limits people working on the same service. Without the ability to run parts of the project in isolation you will have to run all tests for all operations on the service to see if the latest build fixed the remaining defects. This takes time and retesting one test could take you an hour waiting for them all to complete.

Finally your metrics are reduced to the service passing or failing. It’s clear cut but doesn’t really show what’s going on.

Other Considerations

One important factor to consider is how many service testers are going to be testing a single service at a time. If you have multiple then you going to need at least one project per person. The Service Test project files don’t lend themselves to concurrent usage. If you’ve got more resources than services then you will have to consider a structure that is going to maximise resource usage.