How to use Boost Test for automated testing

Content

  1. Introduction
  2. Software you need
  3. Automake/Autoconf stuff
  4. Writing a test class
  5. Test program
  6. Conclusion

1. Introduction

I was asked to write a short tutorial how to use automated testing within KDE. I started playing around with automated tests some time ago by adding some regression tests to my small KDE Edu application KBruch.

I think I don't have to mention the value of automated testing. By adding some test cases to KBruch I was able to identify 3 major bugs. So regression testing is really worth the effort.

Back to the top... |

2. Software you need

I have decided for me to use the Boost testing library. I think the hardest problem to start using automated test is to install Boost. Hopefully you will find some packages for your distribution. Unfortunally, Boost is a very big library and we only need a small part of it. There are many test libraries out there. I am using Boost, because it is well known in the OpenSource community and it is well maintained. Nevertheless, if the KDE project ever decides (and I support this!) to add regression testing in the whole project, it might be a good idea to implement an own testing framework tailored to the actual needs of the KDE project.

Please also take a look at the Boost Test online documentation. There you will find a description of all library functions I use.

Back to the top... |

3. Automake/Autoconf stuff

I will explain the use of the Boost stuff by showing how I implemented test cases for KBruch. So the best thing for you would be to checkout the source from the KDE Edu CVS module. If you don't want to do this or don't know how to do this, you can also follow the next steps by looking at the files in the online CVS service.

First of all I have created a new directory called testcases where I have put the test cases into. Now I added symbolic links from all classes I like to test to the testcases directory. I like to provide a small script for this purpose. Nevertheless, check the results afterwards. And of course this script will fail, if the cpp/h files are located in subdirectories under the ../src directory. But here we go:

#!/bin/bash

# creates symlinks of all cpp/h files in ../src to this directory
#
# Author: Sebastian Stein <seb.kde-AT-hpfsc.de>

# save the current directory
#
current=`pwd`

# delete all symbolic links
#
rm `find . -type l`

# now go to the source directory
#
cd ../src

echo "Creating symlinks to the source files..."

# make a symlink from every *.cpp file to this directory
#
for a in `ls *.cpp`
do
   ln -s ../src/$a ../testcases/$a
done

# make a symlink from every *.h file to this directory
#
for a in `ls *.h`
do
   ln -s ../src/$a ../testcases/$a
done

# and back to the testcases directory
#
cd $current
Now, you also have to edit the Makefiles in the base directory of your project. Just add the new testcases directory to the "SUBDIRS" variable in the Makefile.am file.

The testcases subdirectory also needs a Makefile.am. This one is a little bit harder. Let's take a look at it:

check_PROGRAMS = kbruch_test

kbruch_test_SOURCES = task.cpp ratio.cpp ratio_test.cpp primenumber.cpp primenumber_test.cpp kbruch_test.cpp

kbruch_test_LDADD   = -lboost_unit_test_framework-gcc $(LIB_KDEUI)

# the library search path.
kbruch_test_LDFLAGS = $(all_libraries) $(KDE_RPATH)

EXTRA_DIST = primenumber.cpp primenumber.h ratio.cpp ratio.h task.cpp task.h primenumber_test.cpp ratio_test.cpp kbruch_test.cpp

# set the include path for X, qt and KDE
INCLUDES= $(all_includes)
Before I begin explaining the file please note, I'm not an automake/autoconf guru, so I might just state nonsense... In the first line we add the target. The test will be a program which we have to call to do the tests. This program will be called kbruch_test and can be build by calling make check. Of course we have to say which are the source files of this program. Here you should add all classes you wish to test (e.g. ratio.cpp), the actual test classes (e.g. ratio_test.cpp) and of course the file containing the "main" function of your test program (here kbruch_test.cpp).

All other lines are as usual. The only difference can be found in the LDADD line. You have to add the Boost unit testing framework library here. Unfortunally I can't say how this line must look like for your system. Because the library name contains some system specific information like compiler and multi threaded version or not. Thomas Porschberg has created an automake macro for this purpose. Checkout his great work!

Now we have set up the automake system. Don't forget to run make -f Makefile.cvs and sh configure --enable-debug=yes in the module again. You are still able to build your application by calling make, but you can now also build the test cases by calling make check. After the test cases were built, you have to call the test program with testcases/kbruch_test. But let's go on how to write test classes.

Back to the top... |

4. Writing a test class

I have written all test classes inline. This means I have not created header files for the test classes. I think this is ok for this purpose. Also it is important to think about what you can test with automated tests. It is in general not a good idea to test GUIs. For this task you should use another tool like KD Executor. Testing KBruch is easy. Because the classes I tested are just classes implementing mathematical algorithms. This is always easy to test. But it should also be possible to test e.g. a networking class. The idea is the same. The idea behind regression testing is the black box. You feed some input into the class and you compare the output of the class with the expected output. If they differ, the tested failed. That's it!

Now let's take a look at the primenumber_test class:

// for BOOST testing
#include <boost/test/unit_test.hpp>
using boost::unit_test_framework::test_suite;
using boost::unit_test_framework::test_case;

// the class to be tested
#include "primenumber.h"

class primenumber_test
{
   public:

   // constructor
   primenumber_test()
   {
   }

   /** test the prime number algorithm */
   void test_isPrimeNumber()
   {
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(0) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(2) == 1);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(3) == 1);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(4) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(5) == 1);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(6) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(7) == 1);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(8) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(9) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(23) == 1);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(9) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(9) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(6) == 0);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(101) == 1);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(323) == 0); // 17 * 19
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(1001) == 0); // 7 * 143
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(1002) == 0); // 2 * 501
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(3) == 1);
      BOOST_REQUIRE(m_primenumber.isPrimeNumber(2) == 1);
   }

   /** test the get_first() function */
   void test_get_first()
   {
      BOOST_REQUIRE(m_primenumber.get_first() == 2);
   }

   /** test the move and get functions */
   void test_move_get_func()
   {
      m_primenumber.move_first();
      BOOST_REQUIRE(m_primenumber.get_current() == 2);
      BOOST_REQUIRE(m_primenumber.get_next() == 3);

      m_primenumber.move_forward();
      BOOST_REQUIRE(m_primenumber.get_current() == 5);

      m_primenumber.move_back();
      BOOST_REQUIRE(m_primenumber.get_current() == 3);

      unsigned int tmp = m_primenumber.get_last();
      m_primenumber.move_last();
      BOOST_REQUIRE(m_primenumber.get_current() == tmp);

      m_primenumber.move_forward();
      BOOST_REQUIRE(m_primenumber.get_last() != tmp);
   }

   private:

   // instance of primenumber
   primenumber m_primenumber;
};

class primenumber_test_suite : public test_suite
{
   public:

   primenumber_test_suite() : test_suite("primenumber_test_suite")
   {
      // create an instance of the test cases class
      boost::shared_ptr<primenumber_test> instance(new primenumber_test());

      // create the test cases
      test_case* isPrimeNumber_test_case = BOOST_CLASS_TEST_CASE(
                              &primenumber_test::test_isPrimeNumber, instance );
      test_case* get_first_test_case = BOOST_CLASS_TEST_CASE(
                              &primenumber_test::test_get_first, instance );
      test_case* move_get_func_test_case = BOOST_CLASS_TEST_CASE(
                              &primenumber_test::test_move_get_func, instance );

      // add the test cases to the test suite
      add(isPrimeNumber_test_case);
      add(get_first_test_case);
      add(move_get_func_test_case);
   }
};
As you can see, there are actually 2 classes in this file. Class primenumber_test is the real testing class. It defines a test function for each to be tested public function of class primenumber. I'm not sure if you only can test public functions, but I think it is the right way to do it. You are not interested in the details how the function works. The class is just a black box for you. Just a stable interface (public functions) is important, so it is enough (and already a lot of work) to test those functions.

As you can see, there is the BOOST_REQUIRE() stuff. Each line is a test. Each test function is called a test case. A test case can consist of several tests. The test cases are combined into a test suite. This combination is done in the second class called primenumber_test_suite. You don't have to think much about this class. Just do it that way. It is always the same.

I like to mention that it is sometimes a little bit difficult to find good test cases. But I am pretty sure there are web pages out there describing how to do this. So I won't repeat the ideas here again.

Back to the top... |

5. Test program

Now we have test cases combined into test suites. We now have to combine those test suites into the test program. This is pretty easy. I have done this in kbruch_test.cpp:
// for BOOST testing
#include <boost/test/unit_test.hpp>
using boost::unit_test_framework::test_suite;

// the test classes
#include "primenumber_test.cpp"
#include "ratio_test.cpp"

// test program entry point
test_suite* init_unit_test_suite(int /* argc */, char** /* argv */)
{
   // create the top test suite
   std::auto_ptr<test_suite> top_test_suite(BOOST_TEST_SUITE("Master test suite"));

   // add test suites to the top test suite
   top_test_suite->add(new primenumber_test_suite());
   top_test_suite->add(new ratio_test_suite());

   return top_test_suite.release();
}
The first thing you should notice is, that the program entry function is not called main. It is instead called init_unit_test_suite. In this function the 2 test suites are added and the test is executed. If a test fails, the program will show which test failed and stop the execution. So this is very easy as well.

Back to the top... |

6. Conclusion

I think you have seen that it is not a big deal to add automated tests to your project. The real problem is to install the Boost library and to set up the automake/autoconf stuff. The implementation of the test cases is very easy. Of course you can do a lot more stuff with the Boost Test library. I think it can also generate some statistics, but I have not looked into this yet.

Please feel free to send me any hints, advices or corrections regarding this small tutorial!

Back to the top... |


Contact: seb.kde-AT-hpfsc.de (important: replace -AT- by @)
last modification: 04.10.2007
Click here to load navigation frame