1========================== 2Source-based Code Coverage 3========================== 4 5.. contents:: 6 :local: 7 8Introduction 9============ 10 11This document explains how to use clang's source-based code coverage feature. 12It's called "source-based" because it operates on AST and preprocessor 13information directly. This allows it to generate very precise coverage data. 14 15Clang ships two other code coverage implementations: 16 17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the 18 various sanitizers. It can provide up to edge-level coverage. 19 20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo. 21 This is enabled by ``-ftest-coverage`` or ``--coverage``. 22 23From this point onwards "code coverage" will refer to the source-based kind. 24 25The code coverage workflow 26========================== 27 28The code coverage workflow consists of three main steps: 29 30* Compiling with coverage enabled. 31 32* Running the instrumented program. 33 34* Creating coverage reports. 35 36The next few sections work through a complete, copy-'n-paste friendly example 37based on this program: 38 39.. code-block:: cpp 40 41 % cat <<EOF > foo.cc 42 #define BAR(x) ((x) || (x)) 43 template <typename T> void foo(T x) { 44 for (unsigned I = 0; I < 10; ++I) { BAR(I); } 45 } 46 int main() { 47 foo<int>(0); 48 foo<float>(0); 49 return 0; 50 } 51 EOF 52 53Compiling with coverage enabled 54=============================== 55 56To compile code with coverage enabled, pass ``-fprofile-instr-generate 57-fcoverage-mapping`` to the compiler: 58 59.. code-block:: console 60 61 # Step 1: Compile with coverage enabled. 62 % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo 63 64Note that linking together code with and without coverage instrumentation is 65supported. Uninstrumented code simply won't be accounted for in reports. 66 67Running the instrumented program 68================================ 69 70The next step is to run the instrumented program. When the program exits it 71will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE`` 72environment variable. If that variable does not exist, the profile is written 73to ``default.profraw`` in the current directory of the program. If 74``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing 75directory structure will be created. Additionally, the following special 76**pattern strings** are rewritten: 77 78* "%p" expands out to the process ID. 79 80* "%h" expands out to the hostname of the machine running the program. 81 82* "%t" expands out to the value of the ``TMPDIR`` environment variable. On 83 Darwin, this is typically set to a temporary scratch directory. 84 85* "%Nm" expands out to the instrumented binary's signature. When this pattern 86 is specified, the runtime creates a pool of N raw profiles which are used for 87 on-line profile merging. The runtime takes care of selecting a raw profile 88 from the pool, locking it, and updating it before the program exits. If N is 89 not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must 90 be between 1 and 9. The merge pool specifier can only occur once per filename 91 pattern. 92 93* "%c" expands out to nothing, but enables a mode in which profile counter 94 updates are continuously synced to a file. This means that if the 95 instrumented program crashes, or is killed by a signal, perfect coverage 96 information can still be recovered. Continuous mode does not support value 97 profiling for PGO, and is only supported on Darwin at the moment. Support for 98 Linux may be mostly complete but requires testing, and support for Windows 99 may require more extensive changes: please get involved if you are interested 100 in porting this feature. 101 102.. code-block:: console 103 104 # Step 2: Run the program. 105 % LLVM_PROFILE_FILE="foo.profraw" ./foo 106 107Note that continuous mode is also used on Fuchsia where it's the only supported 108mode, but the implementation is different. The Darwin and Linux implementation 109relies on padding and the ability to map a file over the existing memory 110mapping which is generally only available on POSIX systems and isn't suitable 111for other platforms. 112 113On Fuchsia, we rely on the ability to relocate counters at runtime using a 114level of indirection. On every counter access, we add a bias to the counter 115address. This bias is stored in ``__llvm_profile_counter_bias`` symbol that's 116provided by the profile runtime and is initially set to zero, meaning no 117relocation. The runtime can map the profile into memory at arbitrary locations, 118and set bias to the offset between the original and the new counter location, 119at which point every subsequent counter access will be to the new location, 120which allows updating profile directly akin to the continuous mode. 121 122The advantage of this approach is that doesn't require any special OS support. 123The disadvantage is the extra overhead due to additional instructions required 124for each counter access (overhead both in terms of binary size and performance) 125plus duplication of counters (i.e. one copy in the binary itself and another 126copy that's mapped into memory). This implementation can be also enabled for 127other platforms by passing the ``-runtime-counter-relocation`` option to the 128backend during compilation. 129 130.. code-block:: console 131 132 % clang++ -fprofile-instr-generate -fcoverage-mapping -mllvm -runtime-counter-relocation foo.cc -o foo 133 134Creating coverage reports 135========================= 136 137Raw profiles have to be **indexed** before they can be used to generate 138coverage reports. This is done using the "merge" tool in ``llvm-profdata`` 139(which can combine multiple raw profiles and index them at the same time): 140 141.. code-block:: console 142 143 # Step 3(a): Index the raw profile. 144 % llvm-profdata merge -sparse foo.profraw -o foo.profdata 145 146There are multiple different ways to render coverage reports. The simplest 147option is to generate a line-oriented report: 148 149.. code-block:: console 150 151 # Step 3(b): Create a line-oriented coverage report. 152 % llvm-cov show ./foo -instr-profile=foo.profdata 153 154This report includes a summary view as well as dedicated sub-views for 155templated functions and their instantiations. For our example program, we get 156distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If 157``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line 158region counts (even in macro expansions): 159 160.. code-block:: none 161 162 1| 20|#define BAR(x) ((x) || (x)) 163 ^20 ^2 164 2| 2|template <typename T> void foo(T x) { 165 3| 22| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 166 ^22 ^20 ^20^20 167 4| 2|} 168 ------------------ 169 | void foo<int>(int): 170 | 2| 1|template <typename T> void foo(T x) { 171 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 172 | ^11 ^10 ^10^10 173 | 4| 1|} 174 ------------------ 175 | void foo<float>(int): 176 | 2| 1|template <typename T> void foo(T x) { 177 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 178 | ^11 ^10 ^10^10 179 | 4| 1|} 180 ------------------ 181 182If ``--show-branches=count`` and ``--show-expansions`` are also enabled, the 183sub-views will show detailed branch coverage information in addition to the 184region counts: 185 186.. code-block:: none 187 188 ------------------ 189 | void foo<float>(int): 190 | 2| 1|template <typename T> void foo(T x) { 191 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 192 | ^11 ^10 ^10^10 193 | ------------------ 194 | | | 1| 10|#define BAR(x) ((x) || (x)) 195 | | | ^10 ^1 196 | | | ------------------ 197 | | | | Branch (1:17): [True: 9, False: 1] 198 | | | | Branch (1:24): [True: 0, False: 1] 199 | | | ------------------ 200 | ------------------ 201 | | Branch (3:23): [True: 10, False: 1] 202 | ------------------ 203 | 4| 1|} 204 ------------------ 205 206 207To generate a file-level summary of coverage statistics instead of a 208line-oriented report, try: 209 210.. code-block:: console 211 212 # Step 3(c): Create a coverage summary. 213 % llvm-cov report ./foo -instr-profile=foo.profdata 214 Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover 215 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 216 /tmp/foo.cc 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 217 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 218 TOTAL 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 219 220The ``llvm-cov`` tool supports specifying a custom demangler, writing out 221reports in a directory structure, and generating html reports. For the full 222list of options, please refer to the `command guide 223<https://llvm.org/docs/CommandGuide/llvm-cov.html>`_. 224 225A few final notes: 226 227* The ``-sparse`` flag is optional but can result in dramatically smaller 228 indexed profiles. This option should not be used if the indexed profile will 229 be reused for PGO. 230 231* Raw profiles can be discarded after they are indexed. Advanced use of the 232 profile runtime library allows an instrumented program to merge profiling 233 information directly into an existing raw profile on disk. The details are 234 out of scope. 235 236* The ``llvm-profdata`` tool can be used to merge together multiple raw or 237 indexed profiles. To combine profiling data from multiple runs of a program, 238 try e.g: 239 240 .. code-block:: console 241 242 % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata 243 244Exporting coverage data 245======================= 246 247Coverage data can be exported into JSON using the ``llvm-cov export`` 248sub-command. There is a comprehensive reference which defines the structure of 249the exported data at a high level in the llvm-cov source code. 250 251Interpreting reports 252==================== 253 254There are five statistics tracked in a coverage summary: 255 256* Function coverage is the percentage of functions which have been executed at 257 least once. A function is considered to be executed if any of its 258 instantiations are executed. 259 260* Instantiation coverage is the percentage of function instantiations which 261 have been executed at least once. Template functions and static inline 262 functions from headers are two kinds of functions which may have multiple 263 instantiations. This statistic is hidden by default in reports, but can be 264 enabled via the ``-show-instantiation-summary`` option. 265 266* Line coverage is the percentage of code lines which have been executed at 267 least once. Only executable lines within function bodies are considered to be 268 code lines. 269 270* Region coverage is the percentage of code regions which have been executed at 271 least once. A code region may span multiple lines (e.g in a large function 272 body with no control flow). However, it's also possible for a single line to 273 contain multiple code regions (e.g in "return x || y && z"). 274 275* Branch coverage is the percentage of "true" and "false" branches that have 276 been taken at least once. Each branch is tied to individual conditions in the 277 source code that may each evaluate to either "true" or "false". These 278 conditions may comprise larger boolean expressions linked by boolean logical 279 operators. For example, "x = (y == 2) || (z < 10)" is a boolean expression 280 that is comprised of two individual conditions, each of which evaluates to 281 either true or false, producing four total branch outcomes. 282 283Of these five statistics, function coverage is usually the least granular while 284branch coverage is the most granular. 100% branch coverage for a function 285implies 100% region coverage for a function. The project-wide totals for each 286statistic are listed in the summary. 287 288Format compatibility guarantees 289=============================== 290 291* There are no backwards or forwards compatibility guarantees for the raw 292 profile format. Raw profiles may be dependent on the specific compiler 293 revision used to generate them. It's inadvisable to store raw profiles for 294 long periods of time. 295 296* Tools must retain **backwards** compatibility with indexed profile formats. 297 These formats are not forwards-compatible: i.e, a tool which uses format 298 version X will not be able to understand format version (X+k). 299 300* Tools must also retain **backwards** compatibility with the format of the 301 coverage mappings emitted into instrumented binaries. These formats are not 302 forwards-compatible. 303 304* The JSON coverage export format has a (major, minor, patch) version triple. 305 Only a major version increment indicates a backwards-incompatible change. A 306 minor version increment is for added functionality, and patch version 307 increments are for bugfixes. 308 309Impact of llvm optimizations on coverage reports 310================================================ 311 312llvm optimizations (such as inlining or CFG simplification) should have no 313impact on coverage report quality. This is due to the fact that the mapping 314from source regions to profile counters is immutable, and is generated before 315the llvm optimizer kicks in. The optimizer can't prove that profile counter 316instrumentation is safe to delete (because it's not: it affects the profile the 317program emits), and so leaves it alone. 318 319Note that this coverage feature does not rely on information that can degrade 320during the course of optimization, such as debug info line tables. 321 322Using the profiling runtime without static initializers 323======================================================= 324 325By default the compiler runtime uses a static initializer to determine the 326profile output path and to register a writer function. To collect profiles 327without using static initializers, do this manually: 328 329* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared 330 library and executable. When the linker finds a definition of this symbol, it 331 knows to skip loading the object which contains the profiling runtime's 332 static initializer. 333 334* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it 335 once from each instrumented executable. This function parses 336 ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files 337 at that path. To get the same behavior without truncating existing files, 338 pass a filename pattern string to ``void __llvm_profile_set_filename(char 339 *)``. These calls can be placed anywhere so long as they precede all calls 340 to ``__llvm_profile_write_file``. 341 342* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write 343 out a profile. This function returns 0 when it succeeds, and a non-zero value 344 otherwise. Calling this function multiple times appends profile data to an 345 existing on-disk raw profile. 346 347In C++ files, declare these as ``extern "C"``. 348 349Collecting coverage reports for the llvm project 350================================================ 351 352To prepare a coverage report for llvm (and any of its sub-projects), add 353``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the cmake configuration. Raw 354profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html 355report, run ``llvm/utils/prepare-code-coverage-artifact.py``. 356 357To specify an alternate directory for raw profiles, use 358``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use 359``-DLLVM_PROFILE_MERGE_POOL_SIZE``. 360 361Drawbacks and limitations 362========================= 363 364* Prior to version 2.26, the GNU binutils BFD linker is not able link programs 365 compiled with ``-fcoverage-mapping`` in its ``--gc-sections`` mode. Possible 366 workarounds include disabling ``--gc-sections``, upgrading to a newer version 367 of BFD, or using the Gold linker. 368 369* Code coverage does not handle unpredictable changes in control flow or stack 370 unwinding in the presence of exceptions precisely. Consider the following 371 function: 372 373 .. code-block:: cpp 374 375 int f() { 376 may_throw(); 377 return 0; 378 } 379 380 If the call to ``may_throw()`` propagates an exception into ``f``, the code 381 coverage tool may mark the ``return`` statement as executed even though it is 382 not. A call to ``longjmp()`` can have similar effects. 383 384Clang implementation details 385============================ 386 387This section may be of interest to those wishing to understand or improve 388the clang code coverage implementation. 389 390Gap regions 391----------- 392 393Gap regions are source regions with counts. A reporting tool cannot set a line 394execution count to the count from a gap region unless that region is the only 395one on a line. 396 397Gap regions are used to eliminate unnatural artifacts in coverage reports, such 398as red "unexecuted" highlights present at the end of an otherwise covered line, 399or blue "executed" highlights present at the start of a line that is otherwise 400not executed. 401 402Branch regions 403-------------- 404When viewing branch coverage details in source-based file-level sub-views using 405``--show-branches``, it is recommended that users show all macro expansions 406(using option ``--show-expansions``) since macros may contain hidden branch 407conditions. The coverage summary report will always include these macro-based 408boolean expressions in the overall branch coverage count for a function or 409source file. 410 411Branch coverage is not tracked for constant folded branch conditions since 412branches are not generated for these cases. In the source-based file-level 413sub-view, these branches will simply be shown as ``[Folded - Ignored]`` so that 414users are informed about what happened. 415 416Branch coverage is tied directly to branch-generating conditions in the source 417code. Users should not see hidden branches that aren't actually tied to the 418source code. 419 420 421Switch statements 422----------------- 423 424The region mapping for a switch body consists of a gap region that covers the 425entire body (starting from the '{' in 'switch (...) {', and terminating where the 426last case ends). This gap region has a zero count: this causes "gap" areas in 427between case statements, which contain no executable code, to appear uncovered. 428 429When a switch case is visited, the parent region is extended: if the parent 430region has no start location, its start location becomes the start of the case. 431This is used to support switch statements without a ``CompoundStmt`` body, in 432which the switch body and the single case share a count. 433 434For switches with ``CompoundStmt`` bodies, a new region is created at the start 435of each switch case. 436 437Branch regions are also generated for each switch case, including the default 438case. If there is no explicitly defined default case in the source code, a 439branch region is generated to correspond to the implicit default case that is 440generated by the compiler. The implicit branch region is tied to the line and 441column number of the switch statement condition since no source code for the 442implicit case exists. 443