1========================== 2Source-based Code Coverage 3========================== 4 5.. contents:: 6 :local: 7 8Introduction 9============ 10 11This document explains how to use clang's source-based code coverage feature. 12It's called "source-based" because it operates on AST and preprocessor 13information directly. This allows it to generate very precise coverage data. 14 15Clang ships two other code coverage implementations: 16 17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the 18 various sanitizers. It can provide up to edge-level coverage. 19 20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo. 21 This is enabled by ``-ftest-coverage`` or ``--coverage``. 22 23From this point onwards "code coverage" will refer to the source-based kind. 24 25The code coverage workflow 26========================== 27 28The code coverage workflow consists of three main steps: 29 30* Compiling with coverage enabled. 31 32* Running the instrumented program. 33 34* Creating coverage reports. 35 36The next few sections work through a complete, copy-'n-paste friendly example 37based on this program: 38 39.. code-block:: cpp 40 41 % cat <<EOF > foo.cc 42 #define BAR(x) ((x) || (x)) 43 template <typename T> void foo(T x) { 44 for (unsigned I = 0; I < 10; ++I) { BAR(I); } 45 } 46 int main() { 47 foo<int>(0); 48 foo<float>(0); 49 return 0; 50 } 51 EOF 52 53Compiling with coverage enabled 54=============================== 55 56To compile code with coverage enabled, pass ``-fprofile-instr-generate 57-fcoverage-mapping`` to the compiler: 58 59.. code-block:: console 60 61 # Step 1: Compile with coverage enabled. 62 % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo 63 64Note that linking together code with and without coverage instrumentation is 65supported. Uninstrumented code simply won't be accounted for in reports. 66 67Running the instrumented program 68================================ 69 70The next step is to run the instrumented program. When the program exits it 71will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE`` 72environment variable. If that variable does not exist, the profile is written 73to ``default.profraw`` in the current directory of the program. If 74``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing 75directory structure will be created. Additionally, the following special 76**pattern strings** are rewritten: 77 78* "%p" expands out to the process ID. 79 80* "%h" expands out to the hostname of the machine running the program. 81 82* "%t" expands out to the value of the ``TMPDIR`` environment variable. On 83 Darwin, this is typically set to a temporary scratch directory. 84 85* "%Nm" expands out to the instrumented binary's signature. When this pattern 86 is specified, the runtime creates a pool of N raw profiles which are used for 87 on-line profile merging. The runtime takes care of selecting a raw profile 88 from the pool, locking it, and updating it before the program exits. If N is 89 not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must 90 be between 1 and 9. The merge pool specifier can only occur once per filename 91 pattern. 92 93* "%c" expands out to nothing, but enables a mode in which profile counter 94 updates are continuously synced to a file. This means that if the 95 instrumented program crashes, or is killed by a signal, perfect coverage 96 information can still be recovered. Continuous mode does not support value 97 profiling for PGO, and is only supported on Darwin at the moment. Support for 98 Linux may be mostly complete but requires testing, and support for Windows 99 may require more extensive changes: please get involved if you are interested 100 in porting this feature. 101 102.. code-block:: console 103 104 # Step 2: Run the program. 105 % LLVM_PROFILE_FILE="foo.profraw" ./foo 106 107Note that continuous mode is also used on Fuchsia where it's the only supported 108mode, but the implementation is different. The Darwin and Linux implementation 109relies on padding and the ability to map a file over the existing memory 110mapping which is generally only available on POSIX systems and isn't suitable 111for other platforms. 112 113On Fuchsia, we rely on the ability to relocate counters at runtime using a 114level of indirection. On every counter access, we add a bias to the counter 115address. This bias is stored in ``__llvm_profile_counter_bias`` symbol that's 116provided by the profile runtime and is initially set to zero, meaning no 117relocation. The runtime can map the profile into memory at arbitrary locations, 118and set bias to the offset between the original and the new counter location, 119at which point every subsequent counter access will be to the new location, 120which allows updating profile directly akin to the continuous mode. 121 122The advantage of this approach is that doesn't require any special OS support. 123The disadvantage is the extra overhead due to additional instructions required 124for each counter access (overhead both in terms of binary size and performance) 125plus duplication of counters (i.e. one copy in the binary itself and another 126copy that's mapped into memory). This implementation can be also enabled for 127other platforms by passing the ``-runtime-counter-relocation`` option to the 128backend during compilation. 129 130.. code-block:: console 131 132 % clang++ -fprofile-instr-generate -fcoverage-mapping -mllvm -runtime-counter-relocation foo.cc -o foo 133 134Creating coverage reports 135========================= 136 137Raw profiles have to be **indexed** before they can be used to generate 138coverage reports. This is done using the "merge" tool in ``llvm-profdata`` 139(which can combine multiple raw profiles and index them at the same time): 140 141.. code-block:: console 142 143 # Step 3(a): Index the raw profile. 144 % llvm-profdata merge -sparse foo.profraw -o foo.profdata 145 146There are multiple different ways to render coverage reports. The simplest 147option is to generate a line-oriented report: 148 149.. code-block:: console 150 151 # Step 3(b): Create a line-oriented coverage report. 152 % llvm-cov show ./foo -instr-profile=foo.profdata 153 154This report includes a summary view as well as dedicated sub-views for 155templated functions and their instantiations. For our example program, we get 156distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If 157``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line 158region counts (even in macro expansions): 159 160.. code-block:: none 161 162 1| 20|#define BAR(x) ((x) || (x)) 163 ^20 ^2 164 2| 2|template <typename T> void foo(T x) { 165 3| 22| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 166 ^22 ^20 ^20^20 167 4| 2|} 168 ------------------ 169 | void foo<int>(int): 170 | 2| 1|template <typename T> void foo(T x) { 171 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 172 | ^11 ^10 ^10^10 173 | 4| 1|} 174 ------------------ 175 | void foo<float>(int): 176 | 2| 1|template <typename T> void foo(T x) { 177 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 178 | ^11 ^10 ^10^10 179 | 4| 1|} 180 ------------------ 181 182If ``--show-branches=count`` and ``--show-expansions`` are also enabled, the 183sub-views will show detailed branch coverage information in addition to the 184region counts: 185 186.. code-block:: none 187 188 ------------------ 189 | void foo<float>(int): 190 | 2| 1|template <typename T> void foo(T x) { 191 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 192 | ^11 ^10 ^10^10 193 | ------------------ 194 | | | 1| 10|#define BAR(x) ((x) || (x)) 195 | | | ^10 ^1 196 | | | ------------------ 197 | | | | Branch (1:17): [True: 9, False: 1] 198 | | | | Branch (1:24): [True: 0, False: 1] 199 | | | ------------------ 200 | ------------------ 201 | | Branch (3:23): [True: 10, False: 1] 202 | ------------------ 203 | 4| 1|} 204 ------------------ 205 206 207To generate a file-level summary of coverage statistics instead of a 208line-oriented report, try: 209 210.. code-block:: console 211 212 # Step 3(c): Create a coverage summary. 213 % llvm-cov report ./foo -instr-profile=foo.profdata 214 Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover 215 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 216 /tmp/foo.cc 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 217 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 218 TOTAL 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 219 220The ``llvm-cov`` tool supports specifying a custom demangler, writing out 221reports in a directory structure, and generating html reports. For the full 222list of options, please refer to the `command guide 223<https://llvm.org/docs/CommandGuide/llvm-cov.html>`_. 224 225A few final notes: 226 227* The ``-sparse`` flag is optional but can result in dramatically smaller 228 indexed profiles. This option should not be used if the indexed profile will 229 be reused for PGO. 230 231* Raw profiles can be discarded after they are indexed. Advanced use of the 232 profile runtime library allows an instrumented program to merge profiling 233 information directly into an existing raw profile on disk. The details are 234 out of scope. 235 236* The ``llvm-profdata`` tool can be used to merge together multiple raw or 237 indexed profiles. To combine profiling data from multiple runs of a program, 238 try e.g: 239 240 .. code-block:: console 241 242 % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata 243 244Exporting coverage data 245======================= 246 247Coverage data can be exported into JSON using the ``llvm-cov export`` 248sub-command. There is a comprehensive reference which defines the structure of 249the exported data at a high level in the llvm-cov source code. 250 251Interpreting reports 252==================== 253 254There are four statistics tracked in a coverage summary: 255 256* Function coverage is the percentage of functions which have been executed at 257 least once. A function is considered to be executed if any of its 258 instantiations are executed. 259 260* Instantiation coverage is the percentage of function instantiations which 261 have been executed at least once. Template functions and static inline 262 functions from headers are two kinds of functions which may have multiple 263 instantiations. 264 265* Line coverage is the percentage of code lines which have been executed at 266 least once. Only executable lines within function bodies are considered to be 267 code lines. 268 269* Region coverage is the percentage of code regions which have been executed at 270 least once. A code region may span multiple lines (e.g in a large function 271 body with no control flow). However, it's also possible for a single line to 272 contain multiple code regions (e.g in "return x || y && z"). 273 274* Branch coverage is the percentage of "true" and "false" branches that have 275 been taken at least once. Each branch is tied to individual conditions in the 276 source code that may each evaluate to either "true" or "false". These 277 conditions may comprise larger boolean expressions linked by boolean logical 278 operators. For example, "x = (y == 2) || (z < 10)" is a boolean expression 279 that is comprised of two individual conditions, each of which evaluates to 280 either true or false, producing four total branch outcomes. 281 282Of these five statistics, function coverage is usually the least granular while 283branch coverage is the most granular. 100% branch coverage for a function 284implies 100% region coverage for a function. The project-wide totals for each 285statistic are listed in the summary. 286 287Format compatibility guarantees 288=============================== 289 290* There are no backwards or forwards compatibility guarantees for the raw 291 profile format. Raw profiles may be dependent on the specific compiler 292 revision used to generate them. It's inadvisable to store raw profiles for 293 long periods of time. 294 295* Tools must retain **backwards** compatibility with indexed profile formats. 296 These formats are not forwards-compatible: i.e, a tool which uses format 297 version X will not be able to understand format version (X+k). 298 299* Tools must also retain **backwards** compatibility with the format of the 300 coverage mappings emitted into instrumented binaries. These formats are not 301 forwards-compatible. 302 303* The JSON coverage export format has a (major, minor, patch) version triple. 304 Only a major version increment indicates a backwards-incompatible change. A 305 minor version increment is for added functionality, and patch version 306 increments are for bugfixes. 307 308Impact of llvm optimizations on coverage reports 309================================================ 310 311llvm optimizations (such as inlining or CFG simplification) should have no 312impact on coverage report quality. This is due to the fact that the mapping 313from source regions to profile counters is immutable, and is generated before 314the llvm optimizer kicks in. The optimizer can't prove that profile counter 315instrumentation is safe to delete (because it's not: it affects the profile the 316program emits), and so leaves it alone. 317 318Note that this coverage feature does not rely on information that can degrade 319during the course of optimization, such as debug info line tables. 320 321Using the profiling runtime without static initializers 322======================================================= 323 324By default the compiler runtime uses a static initializer to determine the 325profile output path and to register a writer function. To collect profiles 326without using static initializers, do this manually: 327 328* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared 329 library and executable. When the linker finds a definition of this symbol, it 330 knows to skip loading the object which contains the profiling runtime's 331 static initializer. 332 333* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it 334 once from each instrumented executable. This function parses 335 ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files 336 at that path. To get the same behavior without truncating existing files, 337 pass a filename pattern string to ``void __llvm_profile_set_filename(char 338 *)``. These calls can be placed anywhere so long as they precede all calls 339 to ``__llvm_profile_write_file``. 340 341* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write 342 out a profile. This function returns 0 when it succeeds, and a non-zero value 343 otherwise. Calling this function multiple times appends profile data to an 344 existing on-disk raw profile. 345 346In C++ files, declare these as ``extern "C"``. 347 348Collecting coverage reports for the llvm project 349================================================ 350 351To prepare a coverage report for llvm (and any of its sub-projects), add 352``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the cmake configuration. Raw 353profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html 354report, run ``llvm/utils/prepare-code-coverage-artifact.py``. 355 356To specify an alternate directory for raw profiles, use 357``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use 358``-DLLVM_PROFILE_MERGE_POOL_SIZE``. 359 360Drawbacks and limitations 361========================= 362 363* Prior to version 2.26, the GNU binutils BFD linker is not able link programs 364 compiled with ``-fcoverage-mapping`` in its ``--gc-sections`` mode. Possible 365 workarounds include disabling ``--gc-sections``, upgrading to a newer version 366 of BFD, or using the Gold linker. 367 368* Code coverage does not handle unpredictable changes in control flow or stack 369 unwinding in the presence of exceptions precisely. Consider the following 370 function: 371 372 .. code-block:: cpp 373 374 int f() { 375 may_throw(); 376 return 0; 377 } 378 379 If the call to ``may_throw()`` propagates an exception into ``f``, the code 380 coverage tool may mark the ``return`` statement as executed even though it is 381 not. A call to ``longjmp()`` can have similar effects. 382 383Clang implementation details 384============================ 385 386This section may be of interest to those wishing to understand or improve 387the clang code coverage implementation. 388 389Gap regions 390----------- 391 392Gap regions are source regions with counts. A reporting tool cannot set a line 393execution count to the count from a gap region unless that region is the only 394one on a line. 395 396Gap regions are used to eliminate unnatural artifacts in coverage reports, such 397as red "unexecuted" highlights present at the end of an otherwise covered line, 398or blue "executed" highlights present at the start of a line that is otherwise 399not executed. 400 401Branch regions 402-------------- 403When viewing branch coverage details in source-based file-level sub-views using 404``--show-branches``, it is recommended that users show all macro expansions 405(using option ``--show-expansions``) since macros may contain hidden branch 406conditions. The coverage summary report will always include these macro-based 407boolean expressions in the overall branch coverage count for a function or 408source file. 409 410Branch coverage is not tracked for constant folded branch conditions since 411branches are not generated for these cases. In the source-based file-level 412sub-view, these branches will simply be shown as ``[Folded - Ignored]`` so that 413users are informed about what happened. 414 415Branch coverage is tied directly to branch-generating conditions in the source 416code. Users should not see hidden branches that aren't actually tied to the 417source code. 418 419 420Switch statements 421----------------- 422 423The region mapping for a switch body consists of a gap region that covers the 424entire body (starting from the '{' in 'switch (...) {', and terminating where the 425last case ends). This gap region has a zero count: this causes "gap" areas in 426between case statements, which contain no executable code, to appear uncovered. 427 428When a switch case is visited, the parent region is extended: if the parent 429region has no start location, its start location becomes the start of the case. 430This is used to support switch statements without a ``CompoundStmt`` body, in 431which the switch body and the single case share a count. 432 433For switches with ``CompoundStmt`` bodies, a new region is created at the start 434of each switch case. 435 436Branch regions are also generated for each switch case, including the default 437case. If there is no explicitly defined default case in the source code, a 438branch region is generated to correspond to the implicit default case that is 439generated by the compiler. The implicit branch region is tied to the line and 440column number of the switch statement condition since no source code for the 441implicit case exists. 442