Skip to main content

An Introduction to CTest

I've seen a lot of people (I'm looking at you Daniel Lemire) praise newer languages like Go, which makes me sad as a C++ programmer. Well, that's until I realise that most of the features touted could be easily incorporated in C++ with just a bit of elbow grease. In this post, I'll show how to add an automated testing system using CTest.

I love CMake, it allows me to write cross-platform code and be confident that the build system would work across a choice of compilers, IDEs, and operating systems. When writing large projects, however, it is imperative to have a series of tests that could indicate regression bugs whenever new features are added. While I love the testing support built into languages such as Java, writing tests is not all that hard in C++ either. For example, I could simply write a function that mimics a unit test and prints out either "Test passed" or "Test failed" depending on the result of the test. All I now need is a way to automatically run these tests.

This is where CTest comes into the picture. I'm assuming here that you are using CMake as your build system. If you are not, then you are clearly wrong, a terrible person, and the type who would probably not write tests anyway. So stop reading this post. 😄

Okay, back from that little digression.

My setup for testing is thus: I am building a library that defines all the required functions. Each class/function/API must be tested, so I write tests for each of these. The tests are written so that they would indicate success using the keyword "Test passed" or failure using the keyword "Test failed". I put all the tests in a separate tests folder. Within the tests folder, I write my CMakeLists.txt thus.

set(EXECUTABLE_OUTPUT_PATH ${PROJECT_BINARY_DIR}/tests)
set(CTEST_BINARY_DIRECTORY ${PROJECT_BINARY_DIR}/tests)

file(GLOB files "test_*.cc")

foreach(file ${files})
    string(REGEX REPLACE "(^.*/|\\.[^.]*$)" "" file_without_ext ${file})
    add_executable(${file_without_ext} ${file})
    target_link_libraries(${file_without_ext} ${PROJECT_LIBS})
    add_test(${file_without_ext} ${file_without_ext})
    set_tests_properties(${file_without_ext}
        PROPERTIES
        PASS_REGULAR_EXPRESSION "Test passed")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        FAIL_REGULAR_EXPRESSION "(Exception|Test failed)")
    set_tests_properties(${file_without_ext}
        PROPERTIES
        TIMEOUT 120)
endforeach()

This is a really simple script. It loops over all files in the tests folder that match the pattern test_*.cc, i.e. C++ files that start with the test_ prefix. It generates the executable name by stripping out all the leading directories until the path, while also stripping out the extension for the file. It compiles the file and links it against the project libraries. Finally, it tells CMake/CTest that the compiled binary is a test which on passing would have in its output the text "Test passed", and on failure could have the text "Test failed" or "Exception". Finally, I add a timeout of 120 seconds, or two minutes. If the test runs longer than this time, it will be automatically terminated and marked as a failure. This is not ideal -- some of my tests run for just seconds, while the longest test could run for just under a minute. However, the purpose of the timeout here is not to detect performance bugs, rather to prevent the machine running the tests from fritzing out because of a bug that results in the tests doing something really crazy.

Oh, and in order to enable testing, I simply change the CMakeLists.txt on the very top level of the project (the one that defines project name, version, etc.) to include this one extra line.

include(CTest)

Simple, isn't it? Now why would you go running to Go (see what I did there? 😄), when C++ with CMake offers cross platform builds, testing infrastructure, and parallel execution with memory consistency models?

In the next posts on the series, I might just describe how to test for memory leaks using Valgrind/MemCheck or how to test test-coverage (alliteration FTW) using gcov. Stay tuned.

Popular posts from this blog

Progressive Snapshot: Is it worth it?

I turned 25 last year, which in the highly mathematical and calculating eyes of the US insurance industry meant that I had suddenly matured into a much more responsible driver than I was at 24 years and 364 days of age. As a result, I expected my insurance rates to go down. Imagine my surprise when my insurance renewal notice from GEICO actually quoted a $50 increase in my insurance rates. To me, this was a clear signal that it was time to switch companies.Typically, I score really high on brand loyalty. I tend to stick with a brand for as long as possible, unless they really mess up. This qualified as a major mess up. As a result, I started shopping for insurance quotes.Two companies that quoted me significantly lower rates (30%–40% lower) were Progressive and Allstate. Both had an optional programme that could give me further discounts based on my consenting to the companies tracking my driving habits. Now, I am a careful driver – I hardly ever accelerate hard. I hate using the brak…

Build those noise cancelling headphones

So, here's another DIYLet me start by putting the cart before the horse. I shall start with the credits. This project was done while I was working on my Electronics Design Lab, along with my friends, Srujan M and Indrasen Bhattacharya. The work would not have been possible without the generous help received from the staff at Wadhwani Electronics Laboratory, who ensured that the only thing we did right was to leave the lab on time. This project would also not have been possible without the guidance of our dear and learned professors. It would probably have just about become additional dead weight on the head.Enough with the credits, now, I need to dive right into noise cancellation and how it works.The essence of sound is a pressure wave. The pressure wave, when incident on the eardrum sets into motion the complex mechanisms inside the ear, and after a long path, rather like the Cog advertisement, ends up making some nerves vibrate. The nerves send electrical signals to the brain, …

Reading List, December 2017

Brian Merchant, How email open tracking quietly took over the world, in Wired, 11 December 2017. [Online]: https://www.wired.com/story/how-email-open-tracking-quietly-took-over-the-web/It is no longer a secret that every website you visit silently tracks you in an effort to maximise ad revenue. What is less known is that emails also track you, through the use of tracking pixels and redirect links. These techniques were used by spammers and legitimate companies alike when creating newsletters or other mass email, in order to figure out their reach. What’s happening now is that private people are also using these techniques in order to create invisible and intrusive read receipts for email, which is incredibly frustrating from a privacy point of view.My solution to the tracking woes? I only open the plain-text component of email, which gets rid of tracking pixels entirely. Redirect links are harder to beat, and I don’t have a good solution for this.Dan Luu, Computer latency 1977–2017. D…