usnic BTL Unit Testing Information ================================== This document briefly describes the scheme put in place for usnic BTL unit testing. This system was very quickly tossed together and warrants a proper revisiting at some point in the future. There are lots of ways to solve these problems and this is just one of them. Future improvement is welcome. Goals ----- * To enable _unit_ testing of isolated functions or sets of functions. This has all sorts of benefits, including: - greater confidence that corner cases work as expected - faster, lower-stress refactoring in the future - influencing future interfaces to have less implicit coupling/state (unit testing is harder otherwise) * To be able to test *static* functions as well as non-static ones. * To avoid cluttering the normal code base with excessive test-related macros/code. * The tests should be easy to run under Valgrind and GDB to facilitate: - automated leak/memory checking - easy debugging compared to a parallel MPI environment Anti-Goals ---------- * Testing the low level networking API (e.g., libfabric). * Testing inter-process interaction, such as PRTE-related functionality. Constraints ----------- * our unit tests should never perturb a normal build in terms of performance or correctness - also should not affect other non-usNIC developers in any way (don't break Ralph's `make check`) * static functions are difficult to test from outside the same source file Design Notes ------------ * Source files named `X.c` include a header at the end named `test/X_test.h` - Rationale: gives tests access to the static functions in `X.c` - Rationale: keeps `X.c` clutter-free * unit test infrastructure lives in `btl_usnic_test.c` and `btl_usnic_test.h` * unit test functionality is built and enabled by passing `--enable-opal-btl-usnic-unit-tests` to configure - Rationale: default state disables all unit test logic, achieving our "non-interference" goals * The tests are run by a new executable that gets built when unit tests are enabled: `opal_btl_usnic_run_tests`. * Tests are registered at dlopen time via an `__attribute__((__constructor__))` function that is generated by invocations of the `USNIC_REGISTER_TEST` macro. - Rationale: add tests in one spot, no need to centralize the list of tests to run separately from the tests themselves. * Tests only use a simple `check()` macro right now that has `assert`-like semantics. - this could easily be expanded in the future, using the check docs as inspiration: http://check.sourceforge.net/doc/check_html/check_4.html#Convenience-Test-Functions Worthy Future Goals/Features ---------------------------- * Add some mocking capabilities. - could use the preprocessor to replace regular function/macro calls with calls to functions/macros that dispatch to changeable function pointers - will require some reorganization of some of the existing code... * Output test results in a format that Jenkins and other tools can understand. - TAP - jUnit XML * Possibly utilize part or all of an existing unit testing framework, e.g.: - check: http://check.sourceforge.net/ (LGPLed) - cUnit: http://cunit.sourceforge.net/ (unfortunately GPLed...) * Re-examine test grouping and numbering. Right now there's no real concept of "suites" or multiple cases within a single test function. Once the number of tests grows to a certain point it will probably make sense to revisit this decision.