Chdir doesn’t work with UNC paths so we need another option. I already had the SysInternals tools installed so it made sense to use them.

If you want to know what is going on: lr_save_string takes a char* pointer and drops it into a HP Parameter and lr_eval_string returns a char* pointer evaluting items in brackets into their values. It’s much simpler than mucking about with sprintf.

//Determine whether a remote directory exists
// - relies on psexec and xcopy and therefore Windows dependant
// - returns:
//      0 if path found
//      not zero if path is not found
int RemoteDirectoryExists (const char* server, const char* path)
{
    long FileHandle = 0 ;
    int Result = 1 ;

    lr_save_string (server, "__RemoteDirectoryExists_Server") ;

    //create batch file to be run on remote server that tries to chdir to the path
    FileHandle = fopen("dir.bat", "w") ;
    if (!FileHandle)
    {
        lr_error_message ("Unable to create the \"dir.bat\" file.") ;

        return Result ;
    }

    fprintf (FileHandle, "chdir %s", path) ;
    fclose (FileHandle) ;

    //xcopy batch file to remote server
    system (lr_eval_string ("xcopy /Y /Q \"dir.bat\" \\\\{__RemoteDirectoryExists_Server}\\c$\\temp\\*")) ;

    //psexec batch on remote
    Result = system (lr_eval_string ("psexec.exe \\\\{__RemoteDirectoryExists_Server} C:\\temp\\dir.bat")) ;

    return Result ;
}

Fairly simple really. It creates the batch file. Xcopies it to the remote server and then runs it locally via psexec. I use this in my automated deployment tests to check whether the install path has been removed after uninstall.

So I created a new table and the requisite model, controller, view files and was ready to test it when CakePHP claimed my database table didn’t exist. It did and I spent plenty of time checking database connection strings and removing all but the simplest of code. I eventually found this blog post by Eric Simmerman.

Delete the cache database table list and it all works fine. A simple fix and part of the reason for this post is to get Eric’s post closer to number one on a Google search. It wasn’t when I searched.

The answer I want to know is at what point did a cache become the final word? Caches are about performance enhancement and nothing else. You cache data so that you don’t have to go as far to get the answer. Distance is time is performance in computers. If you miss the cache then you go to your level two cache or you go to the persistent storage and get the data. The bottom line is that you keep going till you get an answer.

At no point in a cache do you ever say: “Well I don’t have it so you can bloody well sod off. No, I’m not going to ask around. I can’t see it, my eyes are closed and my fingers are in my ears. LA LA LA LA. ”

The easiest way to see if you have a double instance problem is to check the memory addresses at various stages of the program. If they differ and they shouldn’t then you have a problem. Finding the causes is a little bit harder.

My first thought was to use a data breakpoint which is a handy little feature in some debuggers that throws a breakpoint when an address gets changed. Because these same debuggers always give the same memory addresses, assuming your application is deterministic, you will get notified when those address get written to.

The simpler option is to place a breakpoint in the constructor and then check the call stack in each place your object is being instantiated. This works a treat but has a few caveats. Firstly you need to be working with classes, your class needs a constructor and it is easier to observe if you don’t have a plethora of objects being instantiated.

The data breakpoint has the caveat that your debugger supports it and that it is enabled. For some reason in C# it wasn’t enabled.

If you have any good ideas to add to the list, let me know.

Have you ever been coding away, running tests, coding some more, running more tests and then having a large swathe of your tests fail. I did a few hours ago. I looked at it and thought to myself, nah, that can’t be right.

So I ran it again.

Many fails, not the same either. Most of the problems seem to be establishing the system under test and that impacts the eventual results.

Rollback.

Fail.

What? These tests worked last night.

Run again.

Fail.

Fail.

Fail.

Ok, what is normally the cause of this? Caching! Let’s go and have a look at that. The caching code appears to be working as normal.

What if I comment out all of my tests except one?

That one fails 50% of the time. The same problem again. Failing when trying to establish the system under test. Look at the incredibly simple code and wonder how it could possible result in the scenario that being reported.

1. I empty the database
2. I manually create two rows with two unique identifiers
3. I get an error saying that the database already has that row… ?

Ok, halt execution after the first record creation. Everything is OK. Inputs for second row are correct. Still gets an error.

Hit refresh 50 times (these are web based tests here) in the dire hope I can exorcise the system.

Try it in Internet Explorer.

It works perfectly. No errors, no database issues, 100% passes as I have been getting for the past few weeks.

Go to a second machine and launch Firefox.

It works perfectly. No errors, no database issues, 100% passes as I have been getting for the past few weeks.

What is going on?

Back to dev box. Kill Firefox. Start again.

It works perfectly. No errors, no database issues, 100% passes as I have been getting for the past few weeks.

I hate computers.

What was the problem? I don’t know.

Will this affect the operation of my application in production? I don’t know.

Does this bother me? Yes.

To make matters worse it is the same problem each time. The results come up, a bunch of assertion fails. After looking at the results and the test code for a few minutes I move over to the “integration” code to see what is wrong, all looks good. So I start checking the individual objects and their methods and the problem becomes obvious. I’ve not written the method, hell, I’ve not even written the class yet.

This is what I miss the most in dynamic languages. I’ve gotten use to employing static testing that tells me; you know what, before you run your tests, try finishing the freaking program.

For example:

$this->testAction('/testing/post?company=utCompany', array('return' => 'vars')) ;

will result in:

[url] => /testing/post?company=utCompany

While invoking the url directly via the web browser results in:

[url] => Array
(
[url] => testing/post
[company] => utCompany
)

The solution I found is a bit of a hack. There is meant to be a fix in the pipeline but it’s not in my build and I am not ready to update to RC3 (and I’m not even sure if the fix is in RC3). Either way there is no guarantee this feature will work beyond RC2

If the second testAction parameter includes a named array called ‘url’ then the values will be placed in the $this->params object in the controller. This gives us the same net result as when the controller is directly invoked. This is not documented in the CakePHP manual.

$data = array ('company' => 'utCompany') ;

$result = $this->testAction('/testing/post', array
(
'return' => 'vars',
'method' => 'get',
'url' => $data
)) ;

I originally asked this question on StackOverflow but had to answer it myself. Still I got four achievements out of that one question. Not a bad return on my investment.

The second problem relates to caching. It is summed up in this post by Wil Clouser called The Cache that wouldn’t quit. My problem was that I was taking a snapshot of my table before my query ran so I could validate the results. The query had only run just beforehand so CakePHP went to the cache and got my previously unchanged dataset. This one drove me spare. I always forget the cache when I test because my policy is that unless I am testing the cache, the cache is turned off.

The solution is fairly simple. The query command (of which I use a lot rather than CakePHP’s “automagic”) accepts a second parameter called “false”. When supplied it turns caching off.

This still leaves you with a problem because you want to call methods in your model. Your model can’t default to no-cache otherwise your prod code wouldn’t have any caching. The easiest solution I came up with is the following.

All of your model code inherits from AppModel. Update AppModel with the following property.

class AppModel extends Model
{
var $useCache = 'true' ;
}

Then in your test model override the attribute with a value of false.

class UserTest extends User
{
var $name = 'UserTest';
var $useTable = 'users' ;
var $useDbConfig = 'test';
var $useCache = 'false' ;
}

In between those two snippets is your User model. User inherits from AppModel but has no need to manage caching. When you write a method on your model you will need to add one more chunk of code.

$result = parent::query
(
'SELECT
users.id
FROM
users
WHERE
users.name = \'' . $name . '\'
AND
users.suspended = 0',
$this->useCache
) ;

Note that $this->useCache is used as the second parameter. This then gives you the ability to ensure caching is turned off while your test your model.

Ideally services, due to their nature, should have as much testing as possible automated. Automating all this is possible but like everything the cost to effort ratio needs to be worked out. If the coders and testers work together then there is a solution to maximise the benefits from the effort.

When the coder is testing their service they will test the service internals (unit in isolation, unit integration testing) before moving to the service contract. If their service has dependencies on other services then they will probably mock out those services based on the service contract to effectively isolate the service and test it. With the clever use of mocks, the coder can establish a set of tests to perform some degree of non-functional testing (availability for example).

As they are by themselves these mocks are not being reused by other consumers of the mocked service, this means that each coder will need to mock the service for its own uses. This is fairly common; with some of the cool mocking frameworks out there it is easy to quickly mock out your dependencies.

I propose that the coders build up a shared mock library for use in service testing. Like coffee it has its disadvantages but its advantages extend into the realm of the tester as well.

Mock libraries

One of the biggest problems with a mock library is the need to maintain mocks over the course of development. Not only do unit tests need to be maintained, but mocks as well. The second is the need to manage the versioning of mock services with their versioned service counterparts. Without these two things then the mock library will get out of date and fall into disuse.

On top of this in a large development shop you have the problem of mock discovery, where it is stored in version control, mock ownership. I could wrap it all in the grandiose term Mock Governance but in reality it isn’t that complicated and doesn’t need that much management. It just needs to be thought about before you start.

The benefits of a mock library are such that each service has this framework of code around it to ensure that it works correctly. As you build the service, unit tests are written to make sure the internals work correctly. As its consumers are being built, mocks are being written to ensure that they handle the services behaviour correctly.

A bigger benefit is seen when we move into the established integration testing environments. When the testers get their hand on the service code they are going to try and do two things. Test the service interfaces to see if they work and secondly; test the service integration to make sure that they services work together.

The service integration as mentioned before is the hard part. With many possible combinations of services and a variety of behaviours coming out of each service the permutations can escalate out of control. Naturally not everything can be tested and what can be tested can be made easier.

Automated Integration Testing

If you take a tool like Capistrano and use it to build a suite of deployment scripts. These scripts deploy combinations of real services and mock services automatically. These scripts become your automated integration tests. You can setup your integration environment to simulate the deployment of one service and five mock services to see how it handles the unavailability of its dependencies. You could deploy the real version of all the services to your performance testing environment. Add a mock consumer to simulate the UI layer and you can generate some load on your services to see how it handles.

If you have a virtualisation server sitting idle you can spin up multiple instances of your service integration at the same time. This can reduce the time to test. When testing integration you don’t always need the entire environment replicated for your test, just the nodes that are within the scope of your test.

Why not service testing tools?

Service testing tools basically provide a wrapper UI for a service. This allows the tester to generate test scripts for validating the interface. The coders however, tested that when they tested the service in isolation. The tester should verify the results, but there is no need to duplicate the effort.

The next problem is that if the service has dependencies and they don’t exist, how is a UI wrapper to the service going to help? You are going to find out how well the service handles not having any of its dependencies but you won’t be able to handle any other integration scenarios.

If you ever get to a situation where at least one service in development is lagging behind your integration testing has to wait. Why wait when you can mock and prove that the service behaves under specific scenarios. When you finally get your complete set of services, you can perform full end to end tests using live versions.

It’s not all sour for service testing tools, some tools do handle aspects of load generation and service result verification, but the way I see it’s a lot of money spent and you are not even getting the benefit of automated service integration testing. The complexity of service integration is what makes testing services hard. In comparison service interface testing is trivial.

Why don’t the coders write the Integration tests?

I said that the testers would be writing these inherently code-like scripts. The coders can if they want, there is no reason that they can’t but there are couple of considerations to be kept in mind before making the decision.

The first the skilling issue and whether or not your developers have the necessary skill set and mindset to adequately create the right scripts to provide good test coverage. The second is a resourcing issue. You have N number of developers. Do they have time to write integration tests as well as their unit and service in isolation testing? If they do, good, I’m sure they will get a kick out of writing scripts to spin up virtual machines establish the system under test, let it loose and bring it all down again. It is pretty cool.

A Final Sell

There are other benefits to come all of this. Deployment testing is given a solid workout with each integration test potentially establishing the system under test each time it is run. The automated integration testing scripts can be reused for future work and can be included in a continuous integration server that exercises each new build. It would be nice to know that when I make a code change I can still see whether my service is still integrating with it consumers and dependencies.

The scripts can be used to develop scoped performance, load and stress tests. By creating mocks that enact scenarios of high load (high latency or low throughput) you more easily determine if your load balancer will scale up or out correctly.

In Wikipedia’s defence it does say that the article does not cite any references or sources. This is good a thing because nobody would want to be associated with that statement.

Now I like code coverage. I like statistics, so why wouldn’t I like code coverage. It’s a cool statistic that lets you know some information about the code you are writing and the unit tests that accompany your code. Code coverage can hint to you where additional unit tests should be written. This would be where one type of code coverage is low or even zero.

Code coverage can also give you a false sense of security. This is what worries me. I see people thinking that code coverage is test coverage. Test coverage is the percentage of tests that cover requirements. When you get into permutations and combinations test coverage itself can be a very unrealistic measure of how tested a system truly is.

This is where experienced testers come in to determine what percentage of test cases are on the critical path and what percentage are not. The same applies for unit tests and code coverage. By itself code coverage is a statistic that can easily be misread or abused, with an experienced developer the relevancy of the statistic is brought out and useful unit tests are written.

I will provide an example of where code coverage reports that the world is rosy, but in reality this code wouldn’t pass test. My example is obviously contrived to illustrate my point. In real life a moderately skilled developer would have built a unit test to sufficiently test this code.

Our Code (C++)

Our Unit Test

Our Code Coverage

• Statement Coverage = 100%
• Branch Coverage = 100%

Taking the statistics at face value it would be time to stop working on this code because it cannot be more perfect than this. For those that don’t know, and non-coders may be reading this, there needs to be at least one more unit test where the value of X (ex) is zero. This will cause a divide by zero exception. For completeness one would add the following tests for X (0.0, 1.0, -1.0, a positive and negative value that is less than +/-1.0f and a positive and negative value that is greater than +/-1.0f).

The important thing to take home from all of this is that code coverage, like many other tools, figures, statistics and adages, are designed to impart information. Making an educated decision based upon such information is where you come in.

]]> http://distributedlife.com/blog/2008/09/code-coverage.html/feed 0 http://distributedlife.com/blog/2008/08/psychics.html http://distributedlife.com/blog/2008/08/psychics.html#comments Fri, 22 Aug 2008 16:58:06 +0000 Ryan Boucher http://distributedlife.com/blog/?p=123 Psychics are a special property that can exist on any object in any programming language. To mark an object as psychic, simply include a relationship to another object within your framework, API or system. The compiler/interpreter will ensure that any documentation that is generated from the source code or is written by hand, if you ever wrote documentation, is removed. This is to ensure that anyone attempting to use your code won’t be able to work why your framework doesn’t work in certain scenarios.

A free tip to anyone writing source code anywhere for any reason. If you make a dependency on another object and then silently fallback to reduced functionality when that object is not supplied, write it down. Don’t stop there, formally document in comic sans so that everyone notices the glaring type and takes note. If you really feel like helping people who licence your codebase, document the dependency in the dependant object. The following text is often sufficient:

Feature X on this object is not available unless Object Y exists and is correctly configured.

Note: For all those that ask, this is how I like to spend my Friday nights.