1================================ 2Fuzzing LLVM libraries and tools 3================================ 4 5.. contents:: 6 :local: 7 :depth: 2 8 9Introduction 10============ 11 12The LLVM tree includes a number of fuzzers for various components. These are 13built on top of :doc:`LibFuzzer <LibFuzzer>`. 14 15 16Available Fuzzers 17================= 18 19clang-fuzzer 20------------ 21 22A |generic fuzzer| that tries to compile textual input as C++ code. Some of the 23bugs this fuzzer has reported are `on bugzilla`__ and `on OSS Fuzz's 24tracker`__. 25 26__ https://llvm.org/pr23057 27__ https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-llvm+clang-fuzzer 28 29clang-proto-fuzzer 30------------------ 31 32A |protobuf fuzzer| that compiles valid C++ programs generated from a protobuf 33class that describes a subset of the C++ language. 34 35This fuzzer accepts clang command line options after `ignore_remaining_args=1`. 36For example, the following command will fuzz clang with a higher optimization 37level: 38 39.. code-block:: shell 40 41 % bin/clang-proto-fuzzer <corpus-dir> -ignore_remaining_args=1 -O3 42 43clang-format-fuzzer 44------------------- 45 46A |generic fuzzer| that runs clang-format_ on C++ text fragments. Some of the 47bugs this fuzzer has reported are `on bugzilla`__ 48and `on OSS Fuzz's tracker`__. 49 50.. _clang-format: https://clang.llvm.org/docs/ClangFormat.html 51__ https://llvm.org/pr23052 52__ https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-llvm+clang-format-fuzzer 53 54llvm-as-fuzzer 55-------------- 56 57A |generic fuzzer| that tries to parse text as :doc:`LLVM assembly <LangRef>`. 58Some of the bugs this fuzzer has reported are `on bugzilla`__. 59 60__ https://llvm.org/pr24639 61 62llvm-dwarfdump-fuzzer 63--------------------- 64 65A |generic fuzzer| that interprets inputs as object files and runs 66:doc:`llvm-dwarfdump <CommandGuide/llvm-dwarfdump>` on them. Some of the bugs 67this fuzzer has reported are `on OSS Fuzz's tracker`__ 68 69__ https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj-llvm+llvm-dwarfdump-fuzzer 70 71llvm-isel-fuzzer 72---------------- 73 74A |LLVM IR fuzzer| aimed at finding bugs in instruction selection. 75 76This fuzzer accepts flags after `ignore_remaining_args=1`. The flags match 77those of :doc:`llc <CommandGuide/llc>` and the triple is required. For example, 78the following command would fuzz AArch64 with :doc:`GlobalISel`: 79 80.. code-block:: shell 81 82 % bin/llvm-isel-fuzzer <corpus-dir> -ignore_remaining_args=1 -mtriple aarch64 -global-isel -O0 83 84Some flags can also be specified in the binary name itself in order to support 85OSS Fuzz, which has trouble with required arguments. To do this, you can copy 86or move ``llvm-isel-fuzzer`` to ``llvm-isel-fuzzer--x-y-z``, separating options 87from the binary name using "--". The valid options are architecture names 88(``aarch64``, ``x86_64``), optimization levels (``O0``, ``O2``), or specific 89keywords, like ``gisel`` for enabling global instruction selection. In this 90mode, the same example could be run like so: 91 92.. code-block:: shell 93 94 % bin/llvm-isel-fuzzer--aarch64-O0-gisel <corpus-dir> 95 96llvm-mc-assemble-fuzzer 97----------------------- 98 99A |generic fuzzer| that fuzzes the MC layer's assemblers by treating inputs as 100target specific assembly. 101 102Note that this fuzzer has an unusual command line interface which is not fully 103compatible with all of libFuzzer's features. Fuzzer arguments must be passed 104after ``--fuzzer-args``, and any ``llc`` flags must use two dashes. For 105example, to fuzz the AArch64 assembler you might use the following command: 106 107.. code-block:: console 108 109 llvm-mc-fuzzer --triple=aarch64-linux-gnu --fuzzer-args -max_len=4 110 111This scheme will likely change in the future. 112 113llvm-mc-disassemble-fuzzer 114-------------------------- 115 116A |generic fuzzer| that fuzzes the MC layer's disassemblers by treating inputs 117as assembled binary data. 118 119Note that this fuzzer has an unusual command line interface which is not fully 120compatible with all of libFuzzer's features. See the notes above about 121``llvm-mc-assemble-fuzzer`` for details. 122 123 124.. |generic fuzzer| replace:: :ref:`generic fuzzer <fuzzing-llvm-generic>` 125.. |protobuf fuzzer| 126 replace:: :ref:`libprotobuf-mutator based fuzzer <fuzzing-llvm-protobuf>` 127.. |LLVM IR fuzzer| 128 replace:: :ref:`structured LLVM IR fuzzer <fuzzing-llvm-ir>` 129 130 131Mutators and Input Generators 132============================= 133 134The inputs for a fuzz target are generated via random mutations of a 135:ref:`corpus <libfuzzer-corpus>`. There are a few options for the kinds of 136mutations that a fuzzer in LLVM might want. 137 138.. _fuzzing-llvm-generic: 139 140Generic Random Fuzzing 141---------------------- 142 143The most basic form of input mutation is to use the built in mutators of 144LibFuzzer. These simply treat the input corpus as a bag of bits and make random 145mutations. This type of fuzzer is good for stressing the surface layers of a 146program, and is good at testing things like lexers, parsers, or binary 147protocols. 148 149Some of the in-tree fuzzers that use this type of mutator are `clang-fuzzer`_, 150`clang-format-fuzzer`_, `llvm-as-fuzzer`_, `llvm-dwarfdump-fuzzer`_, 151`llvm-mc-assemble-fuzzer`_, and `llvm-mc-disassemble-fuzzer`_. 152 153.. _fuzzing-llvm-protobuf: 154 155Structured Fuzzing using ``libprotobuf-mutator`` 156------------------------------------------------ 157 158We can use libprotobuf-mutator_ in order to perform structured fuzzing and 159stress deeper layers of programs. This works by defining a protobuf class that 160translates arbitrary data into structurally interesting input. Specifically, we 161use this to work with a subset of the C++ language and perform mutations that 162produce valid C++ programs in order to exercise parts of clang that are more 163interesting than parser error handling. 164 165To build this kind of fuzzer you need `protobuf`_ and its dependencies 166installed, and you need to specify some extra flags when configuring the build 167with :doc:`CMake <CMake>`. For example, `clang-proto-fuzzer`_ can be enabled by 168adding ``-DCLANG_ENABLE_PROTO_FUZZER=ON`` to the flags described in 169:ref:`building-fuzzers`. 170 171The only in-tree fuzzer that uses ``libprotobuf-mutator`` today is 172`clang-proto-fuzzer`_. 173 174.. _libprotobuf-mutator: https://github.com/google/libprotobuf-mutator 175.. _protobuf: https://github.com/google/protobuf 176 177.. _fuzzing-llvm-ir: 178 179Structured Fuzzing of LLVM IR 180----------------------------- 181 182We also use a more direct form of structured fuzzing for fuzzers that take 183:doc:`LLVM IR <LangRef>` as input. This is achieved through the ``FuzzMutate`` 184library, which was `discussed at EuroLLVM 2017`_. 185 186The ``FuzzMutate`` library is used to structurally fuzz backends in 187`llvm-isel-fuzzer`_. 188 189.. _discussed at EuroLLVM 2017: https://www.youtube.com/watch?v=UBbQ_s6hNgg 190 191 192Building and Running 193==================== 194 195.. _building-fuzzers: 196 197Configuring LLVM to Build Fuzzers 198--------------------------------- 199 200Fuzzers will be built and linked to libFuzzer by default as long as you build 201LLVM with sanitizer coverage enabled. You would typically also enable at least 202one sanitizer for the fuzzers to be particularly likely, so the most common way 203to build the fuzzers is by adding the following two flags to your CMake 204invocation: ``-DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=On``. 205 206.. note:: If you have ``compiler-rt`` checked out in an LLVM tree when building 207 with sanitizers, you'll want to specify ``-DLLVM_BUILD_RUNTIME=Off`` 208 to avoid building the sanitizers themselves with sanitizers enabled. 209 210Continuously Running and Finding Bugs 211------------------------------------- 212 213There used to be a public buildbot running LLVM fuzzers continuously, and while 214this did find issues, it didn't have a very good way to report problems in an 215actionable way. Because of this, we're moving towards using `OSS Fuzz`_ more 216instead. 217 218You can browse the `LLVM project issue list`_ for the bugs found by 219`LLVM on OSS Fuzz`_. These are also mailed to the `llvm-bugs mailing 220list`_. 221 222.. _OSS Fuzz: https://github.com/google/oss-fuzz 223.. _LLVM project issue list: 224 https://bugs.chromium.org/p/oss-fuzz/issues/list?q=Proj-llvm 225.. _LLVM on OSS Fuzz: 226 https://github.com/google/oss-fuzz/blob/master/projects/llvm 227.. _llvm-bugs mailing list: 228 http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs 229 230 231Utilities for Writing Fuzzers 232============================= 233 234There are some utilities available for writing fuzzers in LLVM. 235 236Some helpers for handling the command line interface are available in 237``include/llvm/FuzzMutate/FuzzerCLI.h``, including functions to parse command 238line options in a consistent way and to implement standalone main functions so 239your fuzzer can be built and tested when not built against libFuzzer. 240 241There is also some handling of the CMake config for fuzzers, where you should 242use the ``add_llvm_fuzzer`` to set up fuzzer targets. This function works 243similarly to functions such as ``add_llvm_tool``, but they take care of linking 244to LibFuzzer when appropriate and can be passed the ``DUMMY_MAIN`` argument to 245enable standalone testing. 246