1============== 2Testing libc++ 3============== 4 5.. contents:: 6 :local: 7 8Getting Started 9=============== 10 11libc++ uses LIT to configure and run its tests. The primary way to run the 12libc++ tests is by using make check-libcxx. However since libc++ can be used 13in any number of possible configurations it is important to customize the way 14LIT builds and runs the tests. This guide provides information on how to use 15LIT directly to test libc++. 16 17Please see the `Lit Command Guide`_ for more information about LIT. 18 19.. _LIT Command Guide: http://llvm.org/docs/CommandGuide/lit.html 20 21Setting up the Environment 22-------------------------- 23 24After building libc++ you must setup your environment to test libc++ using 25LIT. 26 27#. Create a shortcut to the actual lit executable so that you can invoke it 28 easily from the command line. 29 30 .. code-block:: bash 31 32 $ alias lit='python path/to/llvm/utils/lit/lit.py' 33 34#. Tell LIT where to find your build configuration. 35 36 .. code-block:: bash 37 38 $ export LIBCXX_SITE_CONFIG=path/to/build-libcxx/test/lit.site.cfg 39 40Example Usage 41------------- 42 43Once you have your environment set up and you have built libc++ you can run 44parts of the libc++ test suite by simply running `lit` on a specified test or 45directory. For example: 46 47.. code-block:: bash 48 49 $ cd path/to/src/libcxx 50 $ lit -sv test/std/re # Run all of the std::regex tests 51 $ lit -sv test/std/depr/depr.c.headers/stdlib_h.pass.cpp # Run a single test 52 $ lit -sv test/std/atomics test/std/threads # Test std::thread and std::atomic 53 54Sometimes you'll want to change the way LIT is running the tests. Custom options 55can be specified using the `--param=<name>=<val>` flag. The most common option 56you'll want to change is the standard dialect (ie -std=c++XX). By default the 57test suite will select the newest C++ dialect supported by the compiler and use 58that. However if you want to manually specify the option like so: 59 60.. code-block:: bash 61 62 $ lit -sv test/std/containers # Run the tests with the newest -std 63 $ lit -sv --param=std=c++03 test/std/containers # Run the tests in C++03 64 65Occasionally you'll want to add extra compile or link flags when testing. 66You can do this as follows: 67 68.. code-block:: bash 69 70 $ lit -sv --param=compile_flags='-Wcustom-warning' 71 $ lit -sv --param=link_flags='-L/custom/library/path' 72 73Some other common examples include: 74 75.. code-block:: bash 76 77 # Specify a custom compiler. 78 $ lit -sv --param=cxx_under_test=/opt/bin/g++ test/std 79 80 # Enable warnings in the test suite 81 $ lit -sv --param=enable_warnings=true test/std 82 83 # Use UBSAN when running the tests. 84 $ lit -sv --param=use_sanitizer=Undefined 85 86 87LIT Options 88=========== 89 90:program:`lit` [*options*...] [*filenames*...] 91 92Command Line Options 93-------------------- 94 95To use these options you pass them on the LIT command line as --param NAME or 96--param NAME=VALUE. Some options have default values specified during CMake's 97configuration. Passing the option on the command line will override the default. 98 99.. program:: lit 100 101.. option:: --cxx_under_test=<path/to/compiler> 102 103 Specify the compiler used to build the tests. 104 105.. option:: --cxx_stdlib_under_test=[libc++, libstdc++, cxx_default] 106 107 Specify the C++ standard library being tested. Unless otherwise specified 108 libc++ is used. This option is intended to allow running the libc++ test 109 suite against other standard library implementations. 110 111.. option:: std=<standard version> 112 113 **Values**: c++98, c++03, c++11, c++14, c++1z 114 115 Change the standard version used when building the tests. 116 117.. option:: --libcxx_site_config=<path/to/lit.site.cfg> 118 119 Specify the site configuration to use when running the tests. This option 120 overrides the enviroment variable LIBCXX_SITE_CONFIG. 121 122.. option:: --cxx_headers=<path/to/headers> 123 124 Specify the c++ standard library headers that are tested. By default the 125 headers in the source tree are used. 126 127.. option:: --cxx_library_root=<path/to/lib/> 128 129 Specify the directory of the libc++ library to be tested. By default the 130 library folder of the build directory is used. This option cannot be used 131 when use_system_lib is provided. 132 133 134.. option:: --cxx_runtime_root=<path/to/lib/> 135 136 Specify the directory of the libc++ library to use at runtime. This directory 137 is not added to the linkers search path. This can be used to compile tests 138 against one version of libc++ and run them using another. The default value 139 for this option is `cxx_library_root`. This option cannot be used 140 when use_system_lib is provided. 141 142.. option:: --use_system_lib=<bool> 143 144 **Default**: False 145 146 Enable or disable testing against the installed version of libc++ library. 147 Note: This does not use the installed headers. 148 149.. option:: --use_lit_shell=<bool> 150 151 Enable or disable the use of LIT's internal shell in ShTests. If the 152 environment variable LIT_USE_INTERNAL_SHELL is present then that is used as 153 the default value. Otherwise the default value is True on Windows and False 154 on every other platform. 155 156.. option:: --no_default_flags=<bool> 157 158 **Default**: False 159 160 Disable all default compile and link flags from being added. When this 161 option is used only flags specified using the compile_flags and link_flags 162 will be used. 163 164.. option:: --compile_flags="<list-of-args>" 165 166 Specify additional compile flags as a space delimited string. 167 Note: This options should not be used to change the standard version used. 168 169.. option:: --link_flags="<list-of-args>" 170 171 Specify additional link flags as a space delimited string. 172 173.. option:: --debug_level=<level> 174 175 **Values**: 0, 1 176 177 Enable the use of debug mode. Level 0 enables assertions and level 1 enables 178 assertions and debugging of iterator misuse. 179 180.. option:: use_sanitizer=<sanitizer name> 181 182 **Values**: Memory, MemoryWithOrigins, Address, Undefined 183 184 Run the tests using the given sanitizer. If LLVM_USE_SANITIZER was given when 185 building libc++ then that sanitizer will be used by default. 186 187.. option:: color_diagnostics 188 189 Enable the use of colorized compile diagnostics. If the color_diagnostics 190 option is specified or the environment variable LIBCXX_COLOR_DIAGNOSTICS is 191 present then color diagnostics will be enabled. 192 193 194Environment Variables 195--------------------- 196 197.. envvar:: LIBCXX_SITE_CONFIG=<path/to/lit.site.cfg> 198 199 Specify the site configuration to use when running the tests. 200 Also see `libcxx_site_config`. 201 202.. envvar:: LIBCXX_COLOR_DIAGNOSTICS 203 204 If ``LIBCXX_COLOR_DIAGNOSTICS`` is defined then the test suite will attempt 205 to use color diagnostic outputs from the compiler. 206 Also see `color_diagnostics`. 207 208Benchmarks 209========== 210 211Libc++ contains benchmark tests separately from the test of the test suite. 212The benchmarks are written using the `Google Benchmark`_ library, a copy of which 213is stored in the libc++ repository. 214 215For more information about using the Google Benchmark library see the 216`official documentation <https://github.com/google/benchmark>`_. 217 218.. _`Google Benchmark`: https://github.com/google/benchmark 219 220Building Benchmarks 221------------------- 222 223The benchmark tests are not built by default. The benchmarks can be built using 224the ``cxx-benchmarks`` target. 225 226An example build would look like: 227 228.. code-block:: bash 229 230 $ cd build 231 $ cmake [options] <path to libcxx sources> 232 $ make cxx-benchmarks 233 234This will build all of the benchmarks under ``<libcxx-src>/benchmarks`` to be 235built against the just-built libc++. The compiled tests are output into 236``build/benchmarks``. 237 238The benchmarks can also be built against the platforms native standard library 239using the ``-DLIBCXX_BUILD_BENCHMARKS_NATIVE_STDLIB=ON`` CMake option. This 240is useful for comparing the performance of libc++ to other standard libraries. 241The compiled benchmarks are named ``<test>.libcxx.out`` if they test libc++ and 242``<test>.native.out`` otherwise. 243 244Also See: 245 246 * :ref:`Building Libc++ <build instructions>` 247 * :ref:`CMake Options` 248 249Running Benchmarks 250------------------ 251 252The benchmarks must be run manually by the user. Currently there is no way 253to run them as part of the build. 254 255For example: 256 257.. code-block:: bash 258 259 $ cd build/benchmarks 260 $ make cxx-benchmarks 261 $ ./algorithms.libcxx.out # Runs all the benchmarks 262 $ ./algorithms.libcxx.out --benchmark_filter=BM_Sort.* # Only runs the sort benchmarks 263 264For more information about running benchmarks see `Google Benchmark`_. 265