Skip to main content

An Introduction to CTest

I've seen a lot of people (I'm looking at you Daniel Lemire) praise newer languages like Go, which makes me sad as a C++ programmer. Well, that's until I realise that most of the features touted could be easily incorporated in C++ with just a bit of elbow grease. In this post, I'll show how to add an automated testing system using CTest.

I love CMake, it allows me to write cross-platform code and be confident that the build system would work across a choice of compilers, IDEs, and operating systems. When writing large projects, however, it is imperative to have a series of tests that could indicate regression bugs whenever new features are added. While I love the testing support built into languages such as Java, writing tests is not all that hard in C++ either. For example, I could simply write a function that mimics a unit test and prints out either "Test passed" or "Test failed" depending on the result of the test. All I now need is a way to automatically run these tests.

This is where CTest comes into the picture. I'm assuming here that you are using CMake as your build system. If you are not, then you are clearly wrong, a terrible person, and the type who would probably not write tests anyway. So stop reading this post. 😄

Okay, back from that little digression.

My setup for testing is thus: I am building a library that defines all the required functions. Each class/function/API must be tested, so I write tests for each of these. The tests are written so that they would indicate success using the keyword "Test passed" or failure using the keyword "Test failed". I put all the tests in a separate tests folder. Within the tests folder, I write my CMakeLists.txt thus.

set(EXECUTABLE_OUTPUT_PATH ${PROJECT_BINARY_DIR}/tests)
set(CTEST_BINARY_DIRECTORY ${PROJECT_BINARY_DIR}/tests)

file(GLOB files "test_*.cc")

foreach(file ${files})
    string(REGEX REPLACE "(^.*/|\\.[^.]*$)" "" file_without_ext ${file})
    add_executable(${file_without_ext} ${file})
    target_link_libraries(${file_without_ext} ${PROJECT_LIBS})
    add_test(${file_without_ext} ${file_without_ext})
    set_tests_properties(${file_without_ext}
        PROPERTIES
        PASS_REGULAR_EXPRESSION "Test passed")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        FAIL_REGULAR_EXPRESSION "(Exception|Test failed)")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        TIMEOUT 120)
endforeach()

This is a really simple script. It loops over all files in the tests folder that match the pattern test_*.cc, i.e. C++ files that start with the test_ prefix. It generates the executable name by stripping out all the leading directories until the path, while also stripping out the extension for the file. It compiles the file and links it against the project libraries. Finally, it tells CMake/CTest that the compiled binary is a test which on passing would have in its output the text "Test passed", and on failure could have the text "Test failed" or "Exception". Finally, I add a timeout of 120 seconds, or two minutes. If the test runs longer than this time, it will be automatically terminated and marked as a failure. This is not ideal -- some of my tests run for just seconds, while the longest test could run for just under a minute. However, the purpose of the timeout here is not to detect performance bugs, rather to prevent the machine running the tests from fritzing out because of a bug that results in the tests doing something really crazy.

Oh, and in order to enable testing, I simply change the CMakeLists.txt on the very top level of the project (the one that defines project name, version, etc.) to include this one extra line.

include(CTest)

Simple, isn't it? Now why would you go running to Go (see what I did there? 😄), when C++ with CMake offers cross platform builds, testing infrastructure, and parallel execution with memory consistency models?

In the next posts on the series, I might just describe how to test for memory leaks using Valgrind/MemCheck or how to test test-coverage (alliteration FTW) using gcov. Stay tuned.

Popular posts from this blog

Progressive Snapshot: Is it worth it?

I turned 25 last year, which in the highly mathematical and calculating eyes of the US insurance industry meant that I had suddenly matured into a much more responsible driver than I was at 24 years and 364 days of age. As a result, I expected my insurance rates to go down. Imagine my surprise when my insurance renewal notice from GEICO actually quoted a $50 increase in my insurance rates. To me, this was a clear signal that it was time to switch companies.Typically, I score really high on brand loyalty. I tend to stick with a brand for as long as possible, unless they really mess up. This qualified as a major mess up. As a result, I started shopping for insurance quotes.Two companies that quoted me significantly lower rates (30%–40% lower) were Progressive and Allstate. Both had an optional programme that could give me further discounts based on my consenting to the companies tracking my driving habits. Now, I am a careful driver – I hardly ever accelerate hard. I hate using the brak…

Cornell Graduate Students United: At What Cost?

On Monday and Tuesday, we graduate students at Cornell will be voting on whether or not we want to unionise. Actually, scratch that, only graduate students who hold a TA, RA, or GRA appointment can unionise.This is a shitty arrangement, and I will be voting against it.For those of you who are not aware of how graduate school works at Cornell, you could be on one of many appointments.FellowshipA graduate student on a fellowship gets a stipend and tuition paid without associated teaching or research opportunities. Graduate students on a fellowship typically work towards their own theses, but will be excluded from the unionGraduate research assistantshipsA GRA gives a graduate student stipend and tuition without teaching responsibilities. However, this money comes out of a specific project grant, and the students typically work on their own theses. Students on GRAs magically qualify to join the union, whereas there is virtually no difference between a GRA and a fellowship for the most pa…

Reading List, April 2017

Adam Carroll, When money isn’t real: The $10,000 experiment, in TEDxLondonBusinessSchool, 9 July 2015. [Online]: https://youtu.be/_VB39Jo8mAQAdam Carroll presents an interesting point – we have abstracted away money through the use of a number of instruments, such as credit and debit cards, NFC payment systems on our phones, and in-app purchases, when we don’t realise how much we are actually spending. Carroll spends some time showing how his kids, aged 7–11 played monopoly differently when they were playing with real money. He goes on to lay his premise, that financial literacy must be taught to children at a young age, when they should be allowed to fail and learn from their failures at a small scale, not at the hundreds of thousands of dollars when they are in student loan debt and just out of college.Carroll’s talk hit a lot of notes with my own experiences with money, and I’m sure that it would resonate with your experiences as well.Brett Scott, If plastic replaces cash, much tha…