distributedlife

In many of the tests I have seen, the following is not uncommon and this is toned down to fit into an already long blog post:

Test Case: Create a Customer

1. Log onto System

a. Enter username

b. Enter password

c. If username of password is not valid and error message is displayed (“username and password are not correct”)

2. Click Open Customer Module

3. Close Customer Module (File -> Exit)

4. Click Menu -> Modules -> Customer Module

5. Close Customer Module (Dialog Window X Button)

6. Click Open Customer Module

7. Click New Customer

8. Enter Customer Name

9. Click Save

10. Message is displayed (“You need to supply some other field”)

11. Enter Address

12. Click Save

13. Message is displayed (“You need to supply some other field”)

14. Enter Date of Birth

15. Click Save

16. Message id displayed (“Customer created”)

What is the purpose of this test? To ensure that one can create a customer? Is it? It might be… if your test cases have multiple personality disorder.

Is it to provide a smoke test of the customer module? Not quite because it only handles creating a customer and not updating, deleting.

Is it end to end testing the create customer workflow? Maybe, except that it only handles some of the negative conditions and doesn’t actually validate whether the customer record was created properly and only handles some of the fields that can be applied to the customer.

Is it providing good test coverage? No, not even remotely.

Is it a test that adds value to the organisations test suite? No. It provides a false sense of testing coverage. An inexperienced tester can run this test, miss existing bugs and the code will go to production.

Is this test easy to maintain when the inevitable change request comes in? No. There are too many aspects to this test and changing the test when the system changes will be messy. I’ll explain shortly how this test should be organised.

So why do we write tests this way? It gives test management the belief that if their current testers spend the vast majority of their billable hours writing test cases then down the track they are going to get value from using non-skilled testers to iterate over the test cases and quality can be maintained while reducing cost.

That doesn’t work. They say that the largest portion to the cost of a system is in the maintenance of it. Therefore it makes sense to design for maintainability rather than reuse. The same tenet should hold for test documentation as well.

What is wrong with the test? Firstly you have login information in your test. While you’re there, why don’t you put; show up to work, chat with co-workers and turn on your computer, at the start of every test? There is nothing more annoying that having to check off that you have logged into the system every test. When you have 7000 tests for a single system and each one asks you to login, that is stupid. Logging in is a valid test, but it only needs to be done once in the positive case and a few times in the negative case, any more than that is some kind of definition of hell.

Do you really need to state someone is logged in? Not really, if you must do it, write a pre-condition that establishes what should exist before the tester starts. No more, if the tester can’t work out how to log into the system then you have bigger problems.

Next up is the two ways into the Customer Module and the two ways to exit the module. How does this relate to creating a customer? They don’t. Take these two tests and pull them out and keep them as separate, distinct cases.

You can take the third “Click Open Customer Module” out of the test as well. You don’t really need it. You don’t tell the user to close the module, logoff and shutdown their computer at the end of the test. Establish the precondition and let’s hope the tester can workout how to open the customer module. If by chance this is non-trivial then in your pre-condition refer the tester to the existing documentation that already says how to open the module. You’ve managed to explain it to your users somehow. Your tester should be emulating a user. Leverage your existing documentation assets and stop repeating yourself.

Test Case: Create a Customer (version 2.0)

1. Click New Customer

2. Enter Customer Name

3. Click Save

4. Message is displayed (“You need to supply an address”)

5. Enter Address

6. Click Save

7. Message is displayed (“You need to supply a date of birth”)

8. Enter Date of Birth

9. Click Save

10. Message id displayed (“Customer created”)

Ok, that leaves with 10 steps and just about all of it relates to the creation of the customer. The problem here is that the scope of the test is too big. Let’s say that the act of saving a customer after the validation has passed throws an error. You fail the test case in its entirety when 80% of it is correct.

From a management perspective they are not getting accurate metrics on progress.

From a developers perspective you are not providing any real granularity over what the actual problem is. When it comes time for them to analyse the defect they have to repeat all the validation tests that you could have proven independently.

From a testing perspective you have to repeat 80% of your test every time you want to test step 9.

From a maintenance perspective if you add validation to the module, then all your create customer tests need to be updated even though you it may be irrelevant.

The underlying problem here is that the test is not scoped enough. Let’s take the assumption that there are only three validation rules and the module is no more complex that the example test case. “A customer needs an address”, “A customer needs a date of birth” and “A customer needs a name”.

“A customer needs a name?” Who picked that one up? It’s implied in step 2 but there is no guarantee that a message will be displayed. It’s still a validation test. This gives you five to six tests:

1. Create Customer where no fields are supplied

2. Create Customer where name is not supplied

3. Create Customer where address is not supplied

4. Create Customer where date of birth is not supplied

5. Create Customer where all mandatory fields are supplied

6. Create Customer where all fields are supplied (might be identical to 5)

In test case 2-4 the expanded description in the test case should state that the other two fields are to be supplied. This helps us isolate the validation. In cases 5 and 6 the required fields are listed or a reference to the user documentation is made. As always we are trying to write as little unnecessary documentation as possible.

Fleshed out our tests are now:

1. Create Customer where no fields are supplied

a. Click New Customer

b. Click Save

c. Message is displayed (“You need to supply a name”)

2. Create Customer where name is not supplied

a. Click New Customer

b. Enter Address

c. Enter Date of Birth

d. Click Save

e. Message is displayed (“You need to supply a name”)

3. Create Customer where address is not supplied

a. Click New Customer

b. Enter Name

c. Enter Date of Birth

d. Click Save

e. Message is displayed (“You need to supply an address”)

4. Create Customer where date of birth is not supplied

a. Click New Customer

b. Enter Name

c. Enter Address

d. Click Save

e. Message is displayed (“You need to supply a date of birth”)

5. Create Customer where all mandatory fields are supplied

a. Click New Customer

b. Enter Name

c. Enter Address

d. Enter Date of Birth

e. Click Save

f. Message is displayed (“Customer created”)

g. Validate customer record was created correctly.

6. Create Customer where all fields are supplied (might be identical to 5)

a. In this example; this is not required.

At this point you may be asking the question. You went from 16 steps to 10 steps then to 5 test cases and a total of 24 test steps. I certainly did. I also increased your test coverage from stuff all to pretty good. Each of the new test cases have no more than 7 steps which is a tiny bite sized chunk of testing that can be completed easily. If we add more validation to the module then we only need to change the relevant test cases (and probably add one or two more) but because they are all nicely scoped a change won’t require superfluous documentation changes.

By breaking up the testing scenarios we now know that if we get a defect the exact problem. It’s not buried in an incomplete test case. The other thing we have done is taken all the unnecessary information out of our test cases.

When we have a new tester arrive to the project or the organisation and they get handed this set of test cases they are not going to know about the customer module, which system it is in, nor how to log in. This is fine. Because you’ve hired a tester and not a business user, the tester will know that they need to review user manuals, wiki documentations, SocialTacit which can be included in the preconditions. They don’t need to step through each precondition before they start, they just need know that they have occurred.