Skip to main content

An Introduction to CTest

I've seen a lot of people (I'm looking at you Daniel Lemire) praise newer languages like Go, which makes me sad as a C++ programmer. Well, that's until I realise that most of the features touted could be easily incorporated in C++ with just a bit of elbow grease. In this post, I'll show how to add an automated testing system using CTest.

I love CMake, it allows me to write cross-platform code and be confident that the build system would work across a choice of compilers, IDEs, and operating systems. When writing large projects, however, it is imperative to have a series of tests that could indicate regression bugs whenever new features are added. While I love the testing support built into languages such as Java, writing tests is not all that hard in C++ either. For example, I could simply write a function that mimics a unit test and prints out either "Test passed" or "Test failed" depending on the result of the test. All I now need is a way to automatically run these tests.

This is where CTest comes into the picture. I'm assuming here that you are using CMake as your build system. If you are not, then you are clearly wrong, a terrible person, and the type who would probably not write tests anyway. So stop reading this post. 😄

Okay, back from that little digression.

My setup for testing is thus: I am building a library that defines all the required functions. Each class/function/API must be tested, so I write tests for each of these. The tests are written so that they would indicate success using the keyword "Test passed" or failure using the keyword "Test failed". I put all the tests in a separate tests folder. Within the tests folder, I write my CMakeLists.txt thus.

set(EXECUTABLE_OUTPUT_PATH ${PROJECT_BINARY_DIR}/tests)
set(CTEST_BINARY_DIRECTORY ${PROJECT_BINARY_DIR}/tests)

file(GLOB files "test_*.cc")

foreach(file ${files})
    string(REGEX REPLACE "(^.*/|\\.[^.]*$)" "" file_without_ext ${file})
    add_executable(${file_without_ext} ${file})
    target_link_libraries(${file_without_ext} ${PROJECT_LIBS})
    add_test(${file_without_ext} ${file_without_ext})
    set_tests_properties(${file_without_ext}
        PROPERTIES
        PASS_REGULAR_EXPRESSION "Test passed")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        FAIL_REGULAR_EXPRESSION "(Exception|Test failed)")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        TIMEOUT 120)
endforeach()

This is a really simple script. It loops over all files in the tests folder that match the pattern test_*.cc, i.e. C++ files that start with the test_ prefix. It generates the executable name by stripping out all the leading directories until the path, while also stripping out the extension for the file. It compiles the file and links it against the project libraries. Finally, it tells CMake/CTest that the compiled binary is a test which on passing would have in its output the text "Test passed", and on failure could have the text "Test failed" or "Exception". Finally, I add a timeout of 120 seconds, or two minutes. If the test runs longer than this time, it will be automatically terminated and marked as a failure. This is not ideal -- some of my tests run for just seconds, while the longest test could run for just under a minute. However, the purpose of the timeout here is not to detect performance bugs, rather to prevent the machine running the tests from fritzing out because of a bug that results in the tests doing something really crazy.

Oh, and in order to enable testing, I simply change the CMakeLists.txt on the very top level of the project (the one that defines project name, version, etc.) to include this one extra line.

include(CTest)

Simple, isn't it? Now why would you go running to Go (see what I did there? 😄), when C++ with CMake offers cross platform builds, testing infrastructure, and parallel execution with memory consistency models?

In the next posts on the series, I might just describe how to test for memory leaks using Valgrind/MemCheck or how to test test-coverage (alliteration FTW) using gcov. Stay tuned.

Comments

Popular posts from this blog

On Harry Potter and why I dislike the series

There could not be a better time for this post. There could not have been a worse time for this post. Now that the penultimate movie of the series is out, and my facebook wall filled with people who loved the movie. But this is something I really wanted to say, and I shall say it anyway. Harry Potter is pathetic literature. Now, you must be wondering why I say that. There are many reasons. Firstly, the storyline itself is flawed. When a writer sits down to write anything, he/she must set up some essential rules about what is happening. These rules must remain constant irrespective of how many times he/she changes his/her mind. This is so that the readers are allowed to have some sensibility in what they are reading. In the fourth book, Rowling goes ahead and kills Cedric. Then, at the end of the book, the horseless carriages are there again. Nothing special. We all knew that they are horseless. But then comes the fifth book, and BAM, the horses are actually winged beasts that only thos...

On the Dvorak Simplified Keyboard

This is a post that I have been meaning to write from quite some time. Long hours spent typing code on my computer left my hands fatigued, and left me with a lot of pain in my wrists and fingers. That is when I decided to use the Dvorak. But I have got the same bad habit as Dr. Watson, to tell a story backwards. Of course, you must be wondering what the Dvorak is. The story of keyboards starts with the invention of the typewriter. Christopher Sholes, the inventor of the typewriter, tried with a two row piano style keyboard. But then, he got into many difficulties with the design. Then he finally settled for a four row design. This was similar to the QWERTY layout that most computers and typewriters today possess. The engineers at Remington, to whom Sholes had presented his design modified the layout a little further, and then the QWERTY was born. As typewriters became popular, people got used to the layout, and started practising touch typing, i.e. typing without looking at the keys...

The paradox of government

I'm fascinated by the concept of government, and the paradoxes it presents. On one hand, governments grant us a certain set of rights or liberties. On the other hand, they work to strip us of the very liberties they promise. Now, I don't mean that all governments strip people of liberties, but there are liberal regimes, and there are sufficiently restrictive and dictatorial ones. Both models may have results to show, it does not mean that people in a restrictive regime are unhappy (refer to Dan Dennett's TED talk , where he states that ideas or memes can be dangerous when taken from one part of the world, where they are widespread, and, using the virus analogy, where people are immune to the memes; to a part of the world where they are foreign, where people may not be immune to the memes and where people may get infected). History has shown that people were sufficiently satisfied with autocratic governments with a benevolent dictator, and that people in other parts of the ...