Build your own test framework

One of the most used library, if not the most, in the world of Java programmers is JUnit. It is a test framework (not necessarily of unit tests) that allows one to write tests this way :

This is a really practical tool, but for a lot of developers, that library (like many other developer’s tools) sometimes has a magical and mysterious aspect, where we tell ourselves that we would certainly not be able to build something like this.

I think that, in computer science, one important element is that one should always understand (at least at high-level) how the tools work. One should use these not because they don’t know how to make them, but rather, because they are not willing to rewrite them. In the majority of cases, it is a lot more advantageous to reuse an already existing tool, that has already been used by a lot of developers and that has been thought out, structured and debugged for some time.

So, I set myself to understand a little bit better how one should go to build a test framework.

First of all, what are the elements that compose a framework of this type minimally ? One should :

  • have access to functions that validate the result of the test, like assertTrue or assertEquals
  • be able to run all of the tests of a class, regardless of their name or number.
  • not fail the program if a test is in error (if it raises an exception, for example)

The first that one could do is write a class of tests like they would want a developer that uses our framework to write it.

Here are the things to notice in this code :

  1. Each test matches a method in the class
  2. One has access to (at least) two functions, checkTrue(…) and checkEqual(…) that determine if the test fails or passes
  3. We do not use (for the moment) an annotation @Test like in JUnit. We simply execute the class as a parameter to send to the framework rather than do their computation via an annotation processor.

Create the validation functions

To create our general validation functions, one has to make them static methods that will be importable in the class that contains the tests.

For the verification of a boolean condition :

For the verification of an equality condition :

For the moment, “.” will be written when the test passes and “F” when it fails. Once that is done, the possibility to know if a test fails or not is provided.

Execution of all the tests

For the moment, if one wants to have the result of a test, they have to call directly by its name the function that contains the test. The goal is to execute all the tests of a class, regardless of their number or name.

To do so, we’ll have to use reflection. It consists in inspecting the structure of a class in the process of running the program.

In that case, the interest is to seek all the methods that are members of the class. We can do that this way :

We have now access to an array that contains all the methods that are defined in this class. Since we want to execute all the functions, we loop through the array, after having defined an instance of the class, and execute each method of the instance testsClass by calling the invoke method :

Do not fail if a test fails

Now, with what we have done, if a test raises an exception, it is all the application that fails. What we want is that the error get catched and printed to screen. On the other hand, the tests should continue to run after that.

We will then wrap the call to invoke in a try … catch this way :

That way, we keep the name of the failing test, the cause and the stack trace of the failure.

Error messages and number of tests

When an exception is raised, it is easy to print the name of the test that fails, since we are already looping on it. However, when it is only the final condition that fails, it can be a little bit harder to detect exactly which test fails (because ultimately, it is only once the call to the verification method is done that we know if we need the name of the test or not.)

Then, we will modify the checkTrue method this way, so that it looks in the call stack from which test we come :

Conclusion and execution

Take our initial example, here is the result when running the tests :

Of course, we are far from a complete test framework that is usable in production, but I hope that this blog post gave you a better idea on how test frameworks can be built, without it being black magic. 🙂

A potential improvement of this library would be to make the tests run without a Main class but with an annotation processor, like in JUnit (that uses the @Test annotation to tell if it is a test to execute). Stay tuned for a future blog post on Annotation Processors, which will allow you to make yourself that improvement if you want.

Also, you’ll be able to find on Github the complete example, as of now, in which the class that contains the tests is received as an argument to the framework, which makes the software a lot more generic.