[fuzzer][fuchsia] Close exception channel before exiting.

On Fuchsia, killing or exiting a process that has a thread listening to its own process's debugger exception channel can hang. Zircon may ki

[fuzzer][fuchsia] Close exception channel before exiting.

On Fuchsia, killing or exiting a process that has a thread listening to its own process's debugger exception channel can hang. Zircon may kill all the threads, send a synthetic exceptions to debugger, and wait for the debugger to have received them. This means the thread listening to the debug exception channel may be killed even as Zircon is waiting for that thread to drain the exception channel, and the process can become stuck in a half-dead state.

This situation is "weird" as it only arises when a process is trying to debug itself. Unfortunately, this is exactly the scenario for libFuzzer on Fuchsia: FuzzerUtilFuchsia spawns a crash-handling thread that acts like a debugger in order to be able to rewrite the crashed threads stack and resume them into libFuzzer's usual POSIX signal handlers. In practice, approximately 25% of fuzzers appear to hang on exit, after generating output and artifacts. These processes hang around until the platform is torn done, which is typically a ClusterFuzz VM. Thus, real-world impact has been somewhat mitigated. The issue should still be resolved for local users, though.

This change improves the behavior of exit() in libFuzzer by adding an atexit handler which closes an event shared with the crash handling thread. This signals to the crash handler that it should close the exception channel and be joined before the process actually exits.

Reviewed By: charco

Differential Revision: https://reviews.llvm.org/D109258

show more ...

[libFuzzer] Fix CFI Directives for fuchsia

This commit fixes the CFI directives in the crash trampoline so
libunwind can get a backtrace during a crash.

In order to get a backtrace from a libfuzzer

[libFuzzer] Fix CFI Directives for fuchsia

This commit fixes the CFI directives in the crash trampoline so
libunwind can get a backtrace during a crash.

In order to get a backtrace from a libfuzzer crash in fuchsia, we
resume execution in the crashed thread, forcing it to call the
StaticCrashHandler. We do this by setting a "crash trampoline" that has
all the necessary cfi directives for an unwinder to get full backtrace
for that thread.

Due to a bug in libunwind, it was not possible to restore the RSP
pointer, as it was always set to the call frame address (CFA). The
previous version worked around this issue by setting the CFA to the
value of the stack pointer at the point of the crash.

The bug in libunwind is now fixed[0], so I am correcting the CFI
annotations so that the CFA correctly points to the beginning of the
trampoline's call frame.

[0]: https://reviews.llvm.org/D106626

Reviewed By: mcgrathr

Differential Revision: https://reviews.llvm.org/D106725

show more ...

Revert "Refactor mutation strategies into a standalone library"

This reverts commit 361f742f168de0f0f256802a329c19d081615d0d.

Refactor mutation strategies into a standalone library

This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing s

Refactor mutation strategies into a standalone library

This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.

Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:

* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)

A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D102447

show more ...

Revert "Refactor mutation strategies into a standalone library"

This reverts commit c4a41cd77c15c2905ac74beeec09f8343a65a549 due to
buildbot failure.

Refactor mutation strategies into a standalone library

This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing s

Refactor mutation strategies into a standalone library

This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.

Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:

* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)

A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D102447

show more ...

[crt][fuzzer] Fix up various numeric conversions

Attempting to build a standalone libFuzzer in Fuchsia's default toolchain for the purpose of cross-compiling the unit tests revealed a number of not

[crt][fuzzer] Fix up various numeric conversions

Attempting to build a standalone libFuzzer in Fuchsia's default toolchain for the purpose of cross-compiling the unit tests revealed a number of not-quite-proper type conversions. Fuchsia's toolchain include `-std=c++17` and `-Werror`, among others, leading to many errors like `-Wshorten-64-to-32`, `-Wimplicit-float-conversion`, etc.

Most of these have been addressed by simply making the conversion explicit with a `static_cast`. These typically fell into one of two categories: 1) conversions between types where high precision isn't critical, e.g. the "energy" calculations for `InputInfo`, and 2) conversions where the values will never reach the bits being truncated, e.g. `DftTimeInSeconds` is not going to exceed 136 years.

The major exception to this is the number of features: there are several places that treat features as `size_t`, and others as `uint32_t`. This change makes the decision to cap the features at 32 bits. The maximum value of a feature as produced by `TracePC::CollectFeatures` is roughly:
(NumPCsInPCTables + ValueBitMap::kMapSizeInBits + ExtraCountersBegin() - ExtraCountersEnd() + log2(SIZE_MAX)) * 8

It's conceivable for extremely large targets and/or extra counters that this limit could be reached. This shouldn't break fuzzing, but it will cause certain features to collide and lower the fuzzers overall precision. To address this, this change adds a warning to TracePC::PrintModuleInfo about excessive feature size if it is detected, and recommends refactoring the fuzzer into several smaller ones.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D97992

show more ...

[libFuzzer] Remove InterruptHandler from Fuchsia implementation

As implemented, the `InterruptHandler` thread was spinning trying to
`select()` on a null "stdin", wasting a significant amount of CPU

[libFuzzer] Remove InterruptHandler from Fuchsia implementation

As implemented, the `InterruptHandler` thread was spinning trying to
`select()` on a null "stdin", wasting a significant amount of CPU for no
benefit. As Fuchsia does not have a native concept of stdin (or POSIX
signals), this commit simply removes this feature entirely.

Reviewed By: aarongreen

Differential Revision: https://reviews.llvm.org/D89266

show more ...

[libFuzzer] Fix minimizing timeouts

When one tries to minimize timeouts using -minimize_crash=1,
minimization immediately fails. The following sequence of events is
responsible for this:

[parent] S

[libFuzzer] Fix minimizing timeouts

When one tries to minimize timeouts using -minimize_crash=1,
minimization immediately fails. The following sequence of events is
responsible for this:

[parent] SIGALRM occurs
[parent] read() returns -EINTR (or -ERESTARTSYS according to strace)
[parent] fgets() returns NULL
[parent] ExecuteCommand() closes child's stdout and returns
[child ] SIGALRM occurs
[child ] AlarmCallback() attempts to write "ALARM: ..." to stdout
[child ] Dies with SIGPIPE without calling DumpCurrentUnit()
[parent] Does not see -exact_artifact_path and exits

When minimizing, the timer in parent is not necessary, so fix by not
setting it in this case.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D85359

show more ...

[libFuzzer] Separate platform related macros out from FuzzerDefs.h into FuzzerPlatform.h, and adjust includes in other files.

Summary: This patch separates platform related macros in lib/fuzzer/Fuzz

[libFuzzer] Separate platform related macros out from FuzzerDefs.h into FuzzerPlatform.h, and adjust includes in other files.

Summary: This patch separates platform related macros in lib/fuzzer/FuzzerDefs.h into lib/fuzzer/FuzzerPlatform.h, and use FuzzerPlatform.h where necessary. This separation helps when compiling libFuzzer's interceptor module (under review); an unnecessary include of standard headers (such as string.h) may produce conflicts/ambiguation with the interceptor's declarations/definitions of library functions, which complicates interceptor implementation.

Reviewers: morehouse, hctim

Reviewed By: morehouse

Subscribers: krytarowski, #sanitizers

Tags: #sanitizers

Differential Revision: https://reviews.llvm.org/D83805

show more ...

[Fuzzer] Rename ExecuteCommandWithPopen to ExecuteCommandNon-Fushsia target will keep using popen/pclose implementation. OnFuchsia, Two-args version of `ExecuteCommand` is a simple wrapper of theone-

[Fuzzer] Rename ExecuteCommandWithPopen to ExecuteCommandNon-Fushsia target will keep using popen/pclose implementation. OnFuchsia, Two-args version of `ExecuteCommand` is a simple wrapper of theone-arg version. (Hopefully) Fix D73329 build on Fuchsia.

show more ...

[libFuzzer] Allow discarding output in ExecuteCommand in Fuchsia.

Summary:
This commit modifies the way `ExecuteCommand` works in fuchsia by adding
special logic to handle `/dev/null`.

The FuzzerCo

[libFuzzer] Allow discarding output in ExecuteCommand in Fuchsia.

Summary:
This commit modifies the way `ExecuteCommand` works in fuchsia by adding
special logic to handle `/dev/null`.

The FuzzerCommand interface does not have a way to "discard" the output,
so other parts of the code just set the output file to `getDevNull()`.
The problem is that fuchsia does not have a named file that is
equivalent to `/dev/null`, so opening that file just fails.

This commit detects whether the specified output file is `getDevNull`,
and if that's the case, it will not copy the file descriptor for stdout
in the spawned process.

NOTE that modifying `FuzzerCommand` to add a "discardOutput" function
involves a significant refactor of all the other platforms, as they all
rely on the `toString()` method of `FuzzerCommand`.

This allows libfuzzer in fuchsia to run with `fork=1`, as the merge
process (`FuzzerMerge.cpp`) invoked `ExecuteCommand` with `/dev/null` as the
output.

Reviewers: aarongreen, phosek

Reviewed By: aarongreen

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D72894

show more ...

[libFuzzer] don't use /dev/null for DiscardOuput in Fuchsia.

Summary:

This commit moves the `DiscardOutput` function in FuzzerIO to
FuzzerUtil, so fuchsia can have its own specialized version.

In

[libFuzzer] don't use /dev/null for DiscardOuput in Fuchsia.

Summary:

This commit moves the `DiscardOutput` function in FuzzerIO to
FuzzerUtil, so fuchsia can have its own specialized version.

In fuchsia, accessing `/dev/null` is not supported, and there's nothing
similar to a file that discards everything that is written to it. The
way of doing something similar in fuchsia is by using `fdio_null_create`
and binding that to a file descriptor with `fdio_bind_to_fd`.

This change should fix one of the issues with the `-close_fd_mask` flag
in libfuzzer, in which closing stdout was not working due to
`fopen("/dev/null", "w")` returning `NULL`.

Reviewers: kcc, aarongreen

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D69593

show more ...

[libFuzzer] Fix unwinding for Fuchsia

Summary:
This commit fixes part of the issues with stack unwinding in fuchsia for
arm64 and x86_64. It consists of multiple fixes:

(1) The cfa_offset calculati

[libFuzzer] Fix unwinding for Fuchsia

Summary:
This commit fixes part of the issues with stack unwinding in fuchsia for
arm64 and x86_64. It consists of multiple fixes:

(1) The cfa_offset calculation was wrong, instead of pointing to the
previous stack pointer, it was pointing to the current one. It worked in
most of the cases because the crashing functions already had a
prologue and had their cfa information relative to another register. The
fix consists on adding a constant that can be used to calculate the
crashing function's stack pointer, and base all the cfi information
relative to that offset.

(2) (arm64) Due to errors with the syntax for the dwarf information, most
of the `OP_NUM` macros were not working. The problem was that they were
referred to as `r##NUM` (like `r14`), when it should have been `x##num`
(like `x14`), or even without the x.

(3) (arm64) The link register was being considered a part of the main
registers (`r30`), when in the real struct it has its own field. Given
that the link register is in the same spot in the struct as r[30] would be,
and that C++ doesn't care about anything, the calculation was still correct.

(4) (x86_64) The stack doesn't need to be aligned to 16 bytes when we
jump to the trampoline function, but it needs to be before performing
call instructions. Encoding that logic in cfi information was tricky, so
we decided to make the cfa information relative to `rbp` and align `rsp`.
Note that this could have been done using another register directly,
but it seems cleaner to make a new fake stack frame.

There are some other minor changes like adding a `brk 1` instruction in
arm64 to make sure that we never return to the crash trampoline (similar to
what we do in x86_64).

Sadly this commit does not fix unwinding for all use cases for arm64.
Crashing functions that do not add information related to the return column in
their cfi information will fail to unwind due to a bug in libunwinder.

Reviewers: mcgrathr, jakehehrlich, phosek, kcc, aarongreen

Subscribers: aprantl, kristof.beyls, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D69579

show more ...

[libFuzzer] Don't prefix absolute paths in fuchsia.

The ExecuteCommand function in fuchsia used to prefix the
getOutputFile for each command run with the artifact_prefix flag if
it was available, be

[libFuzzer] Don't prefix absolute paths in fuchsia.

The ExecuteCommand function in fuchsia used to prefix the
getOutputFile for each command run with the artifact_prefix flag if
it was available, because fuchsia components don't have a writable working
directory. However, if a file with a global path is provided, fuchsia
should honor that.

An example of this is using the global /tmp directory to store stuff.
In fuchsia it ended up being translated to data///tmp, whereas we want
to make sure it is using /tmp (which is available to components using the
isolated-temp feature).

To test this I made the change, compiled fuchsia with this toolchain and
ran a fuzzer with the -fork=1 flag (that mode makes use of the /tmp
directory). I also tested that normal fuzzing workflow was not affected
by this.

Author: charco (Marco Vanotti)
Differential Revision: https://reviews.llvm.org/D68774

llvm-svn: 374612

show more ...

[libFuzzer] Remove lazy counters.

Summary: Lazy counters haven't improved performance for large fuzz targets.

Reviewers: kcc

Reviewed By: kcc

Subscribers: llvm-commits

Tags: #llvm

Differential

[libFuzzer] Remove lazy counters.

Summary: Lazy counters haven't improved performance for large fuzz targets.

Reviewers: kcc

Reviewed By: kcc

Subscribers: llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D67476

llvm-svn: 373403

show more ...

[libFuzzer] Always print DSO map on Fuchsia libFuzzer launch

Fuchsia doesn't have /proc/id/maps, so it relies on the kernel logging system
to provide the DSO map to be able to symbolize in the conte

[libFuzzer] Always print DSO map on Fuchsia libFuzzer launch

Fuchsia doesn't have /proc/id/maps, so it relies on the kernel logging system
to provide the DSO map to be able to symbolize in the context of ASLR. The DSO
map is logged automatically on Fuchsia when encountering a crash or writing to
the sanitizer log for the first time in a process. There are several cases
where libFuzzer doesn't encounter a crash, e.g. on timeouts, OOMs, and when
configured to print new PCs as they become covered, to name a few. Therefore,
this change always writes to the sanitizer log on startup to ensure the DSO map
is available in the log.

Author: aarongreen
Differential Revision: https://reviews.llvm.org/D66233

llvm-svn: 372056

show more ...

[libFuzzer] Migrate to the new exception syscalls on Fuchsia

This is part of the transition to the new Fuchsia exception syscalls
signature.

Differential Revision: https://reviews.llvm.org/D63897

[libFuzzer] Migrate to the new exception syscalls on Fuchsia

This is part of the transition to the new Fuchsia exception syscalls
signature.

Differential Revision: https://reviews.llvm.org/D63897

llvm-svn: 364594

show more ...

[libFuzzer] Ignore synthetic exceptions on Fuchsia

Fuchsia has several exceptions which are merely informational and should
not be treated as crashes. This patch puts logic that read from the
except

[libFuzzer] Ignore synthetic exceptions on Fuchsia

Fuchsia has several exceptions which are merely informational and should
not be treated as crashes. This patch puts logic that read from the
exception port and handled the exception in a loop, and ignores
non-exceptions and informational exceptions.

Patch By: aarongreen

Differential Revision: https://reviews.llvm.org/D62226

llvm-svn: 361407

show more ...

[libFuzzer] experimental performance optimization -lazy_counters, off by default. Posix-only for now, tested on Linux

llvm-svn: 352700

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

[lib/fuzzer] Fix logging for Fuchsia

This change fixes two aspects of logging on Fuchsia:

1. When logging to stdout, the output file descriptor should not be
closed in ExecuteCommand, as it prevent

[lib/fuzzer] Fix logging for Fuchsia

This change fixes two aspects of logging on Fuchsia:

1. When logging to stdout, the output file descriptor should not be
closed in ExecuteCommand, as it prevent fdio_spawn_etc from succeeding
in subsequent calls.
2. When logging to a file in anything other than standalone Zircon, the
log file needs to be created in mutable storage. The best indicator
where this is relative to the Fuchsia component will be given by
'-artifact_prefix=...', so save the log file relative to that path.

Patch by: aarongreen
Differential Revision: https://reviews.llvm.org/D52562

llvm-svn: 343607

show more ...

[sanitizer][fuzzer] Transition back to ZX_TIME_INFINITE

Now that all Zircon calls have been transitioned to take time as signed
value, we can transition back to ZX_TIME_INFINITE, undoing the change

[sanitizer][fuzzer] Transition back to ZX_TIME_INFINITE

Now that all Zircon calls have been transitioned to take time as signed
value, we can transition back to ZX_TIME_INFINITE, undoing the change
made in r337802.

Differential Revision: https://reviews.llvm.org/D51266

llvm-svn: 340764

show more ...

[sanitizer][fuzzer] Temporarily transition to ZX_TIME_INFINITE_OLD

This is a preparation for breaking change when all Zircon calls that
take time as an argument will start using signed valued. We wi

[sanitizer][fuzzer] Temporarily transition to ZX_TIME_INFINITE_OLD

This is a preparation for breaking change when all Zircon calls that
take time as an argument will start using signed valued. We will
transition back to ZX_TIME_INFITINE after all the changes to these
symbols are done and become part of the Fuchsia SDK.

Differential Revision: https://reviews.llvm.org/D49694

llvm-svn: 337802

show more ...

[Fuzzer] Improve crash unwinding on Fuchsia

Fuchsia doesn't have signals; instead it expects processes to have a
dedicated exception thread that binds to the process' exception port and
waits for ex

[Fuzzer] Improve crash unwinding on Fuchsia

Fuchsia doesn't have signals; instead it expects processes to have a
dedicated exception thread that binds to the process' exception port and
waits for exception packets to be delivered. On the other hand,
libFuzzer and sanitizer_common use expect to collect crash information
via libunwind from the same thread that caused the exception.

The long term fix is to improve support for remote unwinding in
libunbwind, plumb this through sanitizer_common and libFuzzer, and
handle the exception exclusively on the exception thread. In the
meantime, this revision has the exception thread "resurrect" the
crashing thread by:

* saving its general purpose register state onto the crashing thread's
stack,
* setting the crashing thread's program counter to an assembly trampoline
with the CFI information needed by libunwind, and
* resuming the crashed thread.

Patch By: aarongreen

Differential Revision: https://reviews.llvm.org/D48509

llvm-svn: 337418

show more ...