1========================== 2Source-based Code Coverage 3========================== 4 5.. contents:: 6 :local: 7 8Introduction 9============ 10 11This document explains how to use clang's source-based code coverage feature. 12It's called "source-based" because it operates on AST and preprocessor 13information directly. This allows it to generate very precise coverage data. 14 15Clang ships two other code coverage implementations: 16 17* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the 18 various sanitizers. It can provide up to edge-level coverage. 19 20* gcov - A GCC-compatible coverage implementation which operates on DebugInfo. 21 This is enabled by ``-ftest-coverage`` or ``--coverage``. 22 23From this point onwards "code coverage" will refer to the source-based kind. 24 25The code coverage workflow 26========================== 27 28The code coverage workflow consists of three main steps: 29 30* Compiling with coverage enabled. 31 32* Running the instrumented program. 33 34* Creating coverage reports. 35 36The next few sections work through a complete, copy-'n-paste friendly example 37based on this program: 38 39.. code-block:: cpp 40 41 % cat <<EOF > foo.cc 42 #define BAR(x) ((x) || (x)) 43 template <typename T> void foo(T x) { 44 for (unsigned I = 0; I < 10; ++I) { BAR(I); } 45 } 46 int main() { 47 foo<int>(0); 48 foo<float>(0); 49 return 0; 50 } 51 EOF 52 53Compiling with coverage enabled 54=============================== 55 56To compile code with coverage enabled, pass ``-fprofile-instr-generate 57-fcoverage-mapping`` to the compiler: 58 59.. code-block:: console 60 61 # Step 1: Compile with coverage enabled. 62 % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo 63 64Note that linking together code with and without coverage instrumentation is 65supported. Uninstrumented code simply won't be accounted for in reports. 66 67Running the instrumented program 68================================ 69 70The next step is to run the instrumented program. When the program exits it 71will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE`` 72environment variable. If that variable does not exist, the profile is written 73to ``default.profraw`` in the current directory of the program. If 74``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing 75directory structure will be created. Additionally, the following special 76**pattern strings** are rewritten: 77 78* "%p" expands out to the process ID. 79 80* "%h" expands out to the hostname of the machine running the program. 81 82* "%t" expands out to the value of the ``TMPDIR`` environment variable. On 83 Darwin, this is typically set to a temporary scratch directory. 84 85* "%Nm" expands out to the instrumented binary's signature. When this pattern 86 is specified, the runtime creates a pool of N raw profiles which are used for 87 on-line profile merging. The runtime takes care of selecting a raw profile 88 from the pool, locking it, and updating it before the program exits. If N is 89 not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must 90 be between 1 and 9. The merge pool specifier can only occur once per filename 91 pattern. 92 93* "%c" expands out to nothing, but enables a mode in which profile counter 94 updates are continuously synced to a file. This means that if the 95 instrumented program crashes, or is killed by a signal, perfect coverage 96 information can still be recovered. Continuous mode does not support value 97 profiling for PGO, and is only supported on Darwin at the moment. Support for 98 Linux may be mostly complete but requires testing, and support for Windows 99 may require more extensive changes: please get involved if you are interested 100 in porting this feature. 101 102.. code-block:: console 103 104 # Step 2: Run the program. 105 % LLVM_PROFILE_FILE="foo.profraw" ./foo 106 107Note that continuous mode is also used on Fuchsia where it's the only supported 108mode, but the implementation is different. The Darwin and Linux implementation 109relies on padding and the ability to map a file over the existing memory 110mapping which is generally only available on POSIX systems and isn't suitable 111for other platforms. 112 113On Fuchsia, we rely on the ability to relocate counters at runtime using a 114level of indirection. On every counter access, we add a bias to the counter 115address. This bias is stored in ``__llvm_profile_counter_bias`` symbol that's 116provided by the profile runtime and is initially set to zero, meaning no 117relocation. The runtime can map the profile into memory at arbitrary locations, 118and set bias to the offset between the original and the new counter location, 119at which point every subsequent counter access will be to the new location, 120which allows updating profile directly akin to the continuous mode. 121 122The advantage of this approach is that doesn't require any special OS support. 123The disadvantage is the extra overhead due to additional instructions required 124for each counter access (overhead both in terms of binary size and performance) 125plus duplication of counters (i.e. one copy in the binary itself and another 126copy that's mapped into memory). This implementation can be also enabled for 127other platforms by passing the ``-runtime-counter-relocation`` option to the 128backend during compilation. 129 130.. code-block:: console 131 132 % clang++ -fprofile-instr-generate -fcoverage-mapping -mllvm -runtime-counter-relocation foo.cc -o foo 133 134Creating coverage reports 135========================= 136 137Raw profiles have to be **indexed** before they can be used to generate 138coverage reports. This is done using the "merge" tool in ``llvm-profdata`` 139(which can combine multiple raw profiles and index them at the same time): 140 141.. code-block:: console 142 143 # Step 3(a): Index the raw profile. 144 % llvm-profdata merge -sparse foo.profraw -o foo.profdata 145 146There are multiple different ways to render coverage reports. The simplest 147option is to generate a line-oriented report: 148 149.. code-block:: console 150 151 # Step 3(b): Create a line-oriented coverage report. 152 % llvm-cov show ./foo -instr-profile=foo.profdata 153 154This report includes a summary view as well as dedicated sub-views for 155templated functions and their instantiations. For our example program, we get 156distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If 157``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line 158region counts (even in macro expansions): 159 160.. code-block:: none 161 162 1| 20|#define BAR(x) ((x) || (x)) 163 ^20 ^2 164 2| 2|template <typename T> void foo(T x) { 165 3| 22| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 166 ^22 ^20 ^20^20 167 4| 2|} 168 ------------------ 169 | void foo<int>(int): 170 | 2| 1|template <typename T> void foo(T x) { 171 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 172 | ^11 ^10 ^10^10 173 | 4| 1|} 174 ------------------ 175 | void foo<float>(int): 176 | 2| 1|template <typename T> void foo(T x) { 177 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 178 | ^11 ^10 ^10^10 179 | 4| 1|} 180 ------------------ 181 182If ``--show-branches=count`` and ``--show-expansions`` are also enabled, the 183sub-views will show detailed branch coverage information in addition to the 184region counts: 185 186.. code-block:: none 187 188 ------------------ 189 | void foo<float>(int): 190 | 2| 1|template <typename T> void foo(T x) { 191 | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } 192 | ^11 ^10 ^10^10 193 | ------------------ 194 | | | 1| 10|#define BAR(x) ((x) || (x)) 195 | | | ^10 ^1 196 | | | ------------------ 197 | | | | Branch (1:17): [True: 9, False: 1] 198 | | | | Branch (1:24): [True: 0, False: 1] 199 | | | ------------------ 200 | ------------------ 201 | | Branch (3:23): [True: 10, False: 1] 202 | ------------------ 203 | 4| 1|} 204 ------------------ 205 206 207To generate a file-level summary of coverage statistics instead of a 208line-oriented report, try: 209 210.. code-block:: console 211 212 # Step 3(c): Create a coverage summary. 213 % llvm-cov report ./foo -instr-profile=foo.profdata 214 Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover Branches Missed Branches Cover 215 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 216 /tmp/foo.cc 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 217 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 218 TOTAL 13 0 100.00% 3 0 100.00% 13 0 100.00% 12 2 83.33% 219 220The ``llvm-cov`` tool supports specifying a custom demangler, writing out 221reports in a directory structure, and generating html reports. For the full 222list of options, please refer to the `command guide 223<https://llvm.org/docs/CommandGuide/llvm-cov.html>`_. 224 225A few final notes: 226 227* The ``-sparse`` flag is optional but can result in dramatically smaller 228 indexed profiles. This option should not be used if the indexed profile will 229 be reused for PGO. 230 231* Raw profiles can be discarded after they are indexed. Advanced use of the 232 profile runtime library allows an instrumented program to merge profiling 233 information directly into an existing raw profile on disk. The details are 234 out of scope. 235 236* The ``llvm-profdata`` tool can be used to merge together multiple raw or 237 indexed profiles. To combine profiling data from multiple runs of a program, 238 try e.g: 239 240 .. code-block:: console 241 242 % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata 243 244Exporting coverage data 245======================= 246 247Coverage data can be exported into JSON using the ``llvm-cov export`` 248sub-command. There is a comprehensive reference which defines the structure of 249the exported data at a high level in the llvm-cov source code. 250 251Interpreting reports 252==================== 253 254There are four statistics tracked in a coverage summary: 255 256* Function coverage is the percentage of functions which have been executed at 257 least once. A function is considered to be executed if any of its 258 instantiations are executed. 259 260* Instantiation coverage is the percentage of function instantiations which 261 have been executed at least once. Template functions and static inline 262 functions from headers are two kinds of functions which may have multiple 263 instantiations. 264 265* Line coverage is the percentage of code lines which have been executed at 266 least once. Only executable lines within function bodies are considered to be 267 code lines. 268 269* Region coverage is the percentage of code regions which have been executed at 270 least once. A code region may span multiple lines (e.g in a large function 271 body with no control flow). However, it's also possible for a single line to 272 contain multiple code regions (e.g in "return x || y && z"). 273 274* Branch coverage is the percentage of "true" and "false" branches that have 275 been taken at least once. Each branch is tied to individual conditions in the 276 source code that may each evaluate to either "true" or "false". These 277 conditions may comprise larger boolean expressions linked by boolean logical 278 operators. For example, "x = (y == 2) || (z < 10)" is a boolean expression 279 that is comprised of two individual conditions, each of which evaluates to 280 either true or false, producing four total branch outcomes. 281 282Of these five statistics, function coverage is usually the least granular while 283branch coverage is the most granular. 100% branch coverage for a function 284implies 100% region coverage for a function. The project-wide totals for each 285statistic are listed in the summary. 286 287Format compatibility guarantees 288=============================== 289 290* There are no backwards or forwards compatibility guarantees for the raw 291 profile format. Raw profiles may be dependent on the specific compiler 292 revision used to generate them. It's inadvisable to store raw profiles for 293 long periods of time. 294 295* Tools must retain **backwards** compatibility with indexed profile formats. 296 These formats are not forwards-compatible: i.e, a tool which uses format 297 version X will not be able to understand format version (X+k). 298 299* Tools must also retain **backwards** compatibility with the format of the 300 coverage mappings emitted into instrumented binaries. These formats are not 301 forwards-compatible. 302 303* The JSON coverage export format has a (major, minor, patch) version triple. 304 Only a major version increment indicates a backwards-incompatible change. A 305 minor version increment is for added functionality, and patch version 306 increments are for bugfixes. 307 308Using the profiling runtime without static initializers 309======================================================= 310 311By default the compiler runtime uses a static initializer to determine the 312profile output path and to register a writer function. To collect profiles 313without using static initializers, do this manually: 314 315* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared 316 library and executable. When the linker finds a definition of this symbol, it 317 knows to skip loading the object which contains the profiling runtime's 318 static initializer. 319 320* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it 321 once from each instrumented executable. This function parses 322 ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files 323 at that path. To get the same behavior without truncating existing files, 324 pass a filename pattern string to ``void __llvm_profile_set_filename(char 325 *)``. These calls can be placed anywhere so long as they precede all calls 326 to ``__llvm_profile_write_file``. 327 328* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write 329 out a profile. This function returns 0 when it succeeds, and a non-zero value 330 otherwise. Calling this function multiple times appends profile data to an 331 existing on-disk raw profile. 332 333In C++ files, declare these as ``extern "C"``. 334 335Collecting coverage reports for the llvm project 336================================================ 337 338To prepare a coverage report for llvm (and any of its sub-projects), add 339``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the cmake configuration. Raw 340profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html 341report, run ``llvm/utils/prepare-code-coverage-artifact.py``. 342 343To specify an alternate directory for raw profiles, use 344``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use 345``-DLLVM_PROFILE_MERGE_POOL_SIZE``. 346 347Drawbacks and limitations 348========================= 349 350* Prior to version 2.26, the GNU binutils BFD linker is not able link programs 351 compiled with ``-fcoverage-mapping`` in its ``--gc-sections`` mode. Possible 352 workarounds include disabling ``--gc-sections``, upgrading to a newer version 353 of BFD, or using the Gold linker. 354 355* Code coverage does not handle unpredictable changes in control flow or stack 356 unwinding in the presence of exceptions precisely. Consider the following 357 function: 358 359 .. code-block:: cpp 360 361 int f() { 362 may_throw(); 363 return 0; 364 } 365 366 If the call to ``may_throw()`` propagates an exception into ``f``, the code 367 coverage tool may mark the ``return`` statement as executed even though it is 368 not. A call to ``longjmp()`` can have similar effects. 369 370Clang implementation details 371============================ 372 373This section may be of interest to those wishing to understand or improve 374the clang code coverage implementation. 375 376Gap regions 377----------- 378 379Gap regions are source regions with counts. A reporting tool cannot set a line 380execution count to the count from a gap region unless that region is the only 381one on a line. 382 383Gap regions are used to eliminate unnatural artifacts in coverage reports, such 384as red "unexecuted" highlights present at the end of an otherwise covered line, 385or blue "executed" highlights present at the start of a line that is otherwise 386not executed. 387 388Branch regions 389-------------- 390When viewing branch coverage details in source-based file-level sub-views using 391``--show-branches``, it is recommended that users show all macro expansions 392(using option ``--show-expansions``) since macros may contain hidden branch 393conditions. The coverage summary report will always include these macro-based 394boolean expressions in the overall branch coverage count for a function or 395source file. 396 397Branch coverage is not tracked for constant folded branch conditions since 398branches are not generated for these cases. In the source-based file-level 399sub-view, these branches will simply be shown as ``[Folded - Ignored]`` so that 400users are informed about what happened. 401 402Branch coverage is tied directly to branch-generating conditions in the source 403code. Users should not see hidden branches that aren't actually tied to the 404source code. 405 406 407Switch statements 408----------------- 409 410The region mapping for a switch body consists of a gap region that covers the 411entire body (starting from the '{' in 'switch (...) {', and terminating where the 412last case ends). This gap region has a zero count: this causes "gap" areas in 413between case statements, which contain no executable code, to appear uncovered. 414 415When a switch case is visited, the parent region is extended: if the parent 416region has no start location, its start location becomes the start of the case. 417This is used to support switch statements without a ``CompoundStmt`` body, in 418which the switch body and the single case share a count. 419 420For switches with ``CompoundStmt`` bodies, a new region is created at the start 421of each switch case. 422 423Branch regions are also generated for each switch case, including the default 424case. If there is no explicitly defined default case in the source code, a 425branch region is generated to correspond to the implicit default case that is 426generated by the compiler. The implicit branch region is tied to the line and 427column number of the switch statement condition since no source code for the 428implicit case exists. 429