Skip to main content

An Introduction to CTest

I've seen a lot of people (I'm looking at you Daniel Lemire) praise newer languages like Go, which makes me sad as a C++ programmer. Well, that's until I realise that most of the features touted could be easily incorporated in C++ with just a bit of elbow grease. In this post, I'll show how to add an automated testing system using CTest.

I love CMake, it allows me to write cross-platform code and be confident that the build system would work across a choice of compilers, IDEs, and operating systems. When writing large projects, however, it is imperative to have a series of tests that could indicate regression bugs whenever new features are added. While I love the testing support built into languages such as Java, writing tests is not all that hard in C++ either. For example, I could simply write a function that mimics a unit test and prints out either "Test passed" or "Test failed" depending on the result of the test. All I now need is a way to automatically run these tests.

This is where CTest comes into the picture. I'm assuming here that you are using CMake as your build system. If you are not, then you are clearly wrong, a terrible person, and the type who would probably not write tests anyway. So stop reading this post. 😄

Okay, back from that little digression.

My setup for testing is thus: I am building a library that defines all the required functions. Each class/function/API must be tested, so I write tests for each of these. The tests are written so that they would indicate success using the keyword "Test passed" or failure using the keyword "Test failed". I put all the tests in a separate tests folder. Within the tests folder, I write my CMakeLists.txt thus.

set(EXECUTABLE_OUTPUT_PATH ${PROJECT_BINARY_DIR}/tests)
set(CTEST_BINARY_DIRECTORY ${PROJECT_BINARY_DIR}/tests)

file(GLOB files "test_*.cc")

foreach(file ${files})
    string(REGEX REPLACE "(^.*/|\\.[^.]*$)" "" file_without_ext ${file})
    add_executable(${file_without_ext} ${file})
    target_link_libraries(${file_without_ext} ${PROJECT_LIBS})
    add_test(${file_without_ext} ${file_without_ext})
    set_tests_properties(${file_without_ext}
        PROPERTIES
        PASS_REGULAR_EXPRESSION "Test passed")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        FAIL_REGULAR_EXPRESSION "(Exception|Test failed)")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        TIMEOUT 120)
endforeach()

This is a really simple script. It loops over all files in the tests folder that match the pattern test_*.cc, i.e. C++ files that start with the test_ prefix. It generates the executable name by stripping out all the leading directories until the path, while also stripping out the extension for the file. It compiles the file and links it against the project libraries. Finally, it tells CMake/CTest that the compiled binary is a test which on passing would have in its output the text "Test passed", and on failure could have the text "Test failed" or "Exception". Finally, I add a timeout of 120 seconds, or two minutes. If the test runs longer than this time, it will be automatically terminated and marked as a failure. This is not ideal -- some of my tests run for just seconds, while the longest test could run for just under a minute. However, the purpose of the timeout here is not to detect performance bugs, rather to prevent the machine running the tests from fritzing out because of a bug that results in the tests doing something really crazy.

Oh, and in order to enable testing, I simply change the CMakeLists.txt on the very top level of the project (the one that defines project name, version, etc.) to include this one extra line.

include(CTest)

Simple, isn't it? Now why would you go running to Go (see what I did there? 😄), when C++ with CMake offers cross platform builds, testing infrastructure, and parallel execution with memory consistency models?

In the next posts on the series, I might just describe how to test for memory leaks using Valgrind/MemCheck or how to test test-coverage (alliteration FTW) using gcov. Stay tuned.

Popular posts from this blog

Progressive Snapshot: Is it worth it?

I turned 25 last year, which in the highly mathematical and calculating eyes of the US insurance industry meant that I had suddenly matured into a much more responsible driver than I was at 24 years and 364 days of age. As a result, I expected my insurance rates to go down. Imagine my surprise when my insurance renewal notice from GEICO actually quoted a $50 increase in my insurance rates. To me, this was a clear signal that it was time to switch companies.Typically, I score really high on brand loyalty. I tend to stick with a brand for as long as possible, unless they really mess up. This qualified as a major mess up. As a result, I started shopping for insurance quotes.Two companies that quoted me significantly lower rates (30%–40% lower) were Progressive and Allstate. Both had an optional programme that could give me further discounts based on my consenting to the companies tracking my driving habits. Now, I am a careful driver – I hardly ever accelerate hard. I hate using the brak…

Cornell Graduate Students United: At What Cost?

On Monday and Tuesday, we graduate students at Cornell will be voting on whether or not we want to unionise. Actually, scratch that, only graduate students who hold a TA, RA, or GRA appointment can unionise.This is a shitty arrangement, and I will be voting against it.For those of you who are not aware of how graduate school works at Cornell, you could be on one of many appointments.FellowshipA graduate student on a fellowship gets a stipend and tuition paid without associated teaching or research opportunities. Graduate students on a fellowship typically work towards their own theses, but will be excluded from the unionGraduate research assistantshipsA GRA gives a graduate student stipend and tuition without teaching responsibilities. However, this money comes out of a specific project grant, and the students typically work on their own theses. Students on GRAs magically qualify to join the union, whereas there is virtually no difference between a GRA and a fellowship for the most pa…

De-addicting

I’m on a process to break away from the distractions of social media, primarily Facebook, and spend the time to pursue other interests. This is an interesting experiment, and it has required significant dedication and effort on my part. At the end, though, I’m not certain if I’ll be successful.I realised that I had a problem when Facebook became the most visited sites in my browser. I should have been working, not scrolling through Facebook like a, well, you know what Jerry Seinfeld has to say. I further noticed that I would tend to log on to Facebook every time I was stuck at work, so that my capacity to tackle problems was reduced to only those that involved cracking a walnut shell. Anything more complex, and I would procrastinate and end up on Facebook.What I needed was a swift kick in the arse to get off Facebook and into the real world. It came through the means of my latest project which required all my attention. At this time, I logged out of Facebook from my work computer and …