[libFuzzer] Add missing argument to CrashResistantMerge.

Fixes a build error caused by a bad merge conflict resolution for
https://reviews.llvm.org/D105084.

Redistribute energy for Corpus

I found that the initial corpus allocation of fork mode has certain defects.
I designed a new initial corpus allocation strategy based on size grouping.
This method ca

Redistribute energy for Corpus

I found that the initial corpus allocation of fork mode has certain defects.
I designed a new initial corpus allocation strategy based on size grouping.
This method can give more energy to the small seeds in the corpus and
increase the throughput of the test.

Fuzzbench data (glibfuzzer is -fork_corpus_groups=1):
https://www.fuzzbench.com/reports/experimental/2021-08-05-parallel/index.html

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D105084

show more ...

Greedy set cover implementation of `Merger::Merge`

Extend the existing single-pass algorithm for `Merger::Merge` with an algorithm that gives better results. This new implementation can be used with

Greedy set cover implementation of `Merger::Merge`

Extend the existing single-pass algorithm for `Merger::Merge` with an algorithm that gives better results. This new implementation can be used with a new **set_cover_merge=1** flag.

This greedy set cover implementation gives a substantially smaller final corpus (40%-80% less testcases) while preserving the same features/coverage. At the same time, the execution time penalty is not that significant (+50% for ~1M corpus files and far less for smaller corpora). These results were obtained by comparing several targets with varying size corpora.

Change `Merger::CrashResistantMergeInternalStep` to collect all features from each file and not just unique ones. This is needed for the set cover algorithm to work correctly. The implementation of the algorithm in `Merger::SetCoverMerge` uses a bitvector to store features that are covered by a file while performing the pass. Collisions while indexing the bitvector are ignored similarly to the fuzzer.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D105284

show more ...

[compiler-rt] NFC: Fix trivial typo

Reviewed By: xgupta

Differential Revision: https://reviews.llvm.org/D77457

[libFuzzer] replace Vector/Set with std::vector/std::set. The custom names are not required any more since we now build with a private version of libc++. Fix some of the 81+ character lines. Mechanic

[libFuzzer] replace Vector/Set with std::vector/std::set. The custom names are not required any more since we now build with a private version of libc++. Fix some of the 81+ character lines. Mechanical change, NFC expected.

[libFuzzer] replace Vector/Set with std::vector/std::set.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D107374

show more ...

[crt][fuzzer] Fix up various numeric conversions

Attempting to build a standalone libFuzzer in Fuchsia's default toolchain for the purpose of cross-compiling the unit tests revealed a number of not

[crt][fuzzer] Fix up various numeric conversions

Attempting to build a standalone libFuzzer in Fuchsia's default toolchain for the purpose of cross-compiling the unit tests revealed a number of not-quite-proper type conversions. Fuchsia's toolchain include `-std=c++17` and `-Werror`, among others, leading to many errors like `-Wshorten-64-to-32`, `-Wimplicit-float-conversion`, etc.

Most of these have been addressed by simply making the conversion explicit with a `static_cast`. These typically fell into one of two categories: 1) conversions between types where high precision isn't critical, e.g. the "energy" calculations for `InputInfo`, and 2) conversions where the values will never reach the bits being truncated, e.g. `DftTimeInSeconds` is not going to exceed 136 years.

The major exception to this is the number of features: there are several places that treat features as `size_t`, and others as `uint32_t`. This change makes the decision to cap the features at 32 bits. The maximum value of a feature as produced by `TracePC::CollectFeatures` is roughly:
(NumPCsInPCTables + ValueBitMap::kMapSizeInBits + ExtraCountersBegin() - ExtraCountersEnd() + log2(SIZE_MAX)) * 8

It's conceivable for extremely large targets and/or extra counters that this limit could be reached. This shouldn't break fuzzing, but it will cause certain features to collide and lower the fuzzers overall precision. To address this, this change adds a warning to TracePC::PrintModuleInfo about excessive feature size if it is detected, and recommends refactoring the fuzzer into several smaller ones.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D97992

show more ...

Expand unit tests for fuzzer::Merger

This change adds additional unit tests for fuzzer::Merger::Parse and fuzzer::Merger::Merge in anticipation of additional changes to the merge control file format

Expand unit tests for fuzzer::Merger

This change adds additional unit tests for fuzzer::Merger::Parse and fuzzer::Merger::Merge in anticipation of additional changes to the merge control file format to support cross-process fuzzing.

It modifies the parameter handling of Merge slightly in order to make NewFeatures and NewCov consistent with NewFiles; namely, Merge *replaces* the contents of these output parameters rather than accumulating them (thereby fixing a buggy return value).

This is change 1 of (at least) 18 for cross-process fuzzing support.

Reviewed By: morehouse

Differential Revision: https://reviews.llvm.org/D94506

show more ...

[libFuzzer] Add an option to keep initial seed inputs around.

This patch adds an option "keep_seed" to keep all initial seed inputs in the
corpus. Previously, only the initial seed inputs that find

[libFuzzer] Add an option to keep initial seed inputs around.

This patch adds an option "keep_seed" to keep all initial seed inputs in the
corpus. Previously, only the initial seed inputs that find new coverage were
added to the corpus, and all the other initial inputs were discarded. We
observed in some circumstances that useful initial seed inputs are discarded as
they find no new coverage, even though they contain useful fragments in them
(e.g., SQLITE3 FuzzBench benchmark). This newly added option provides a way to
keeping seed inputs in the corpus for those circumstances. With this patch, and
with -keep_seed=1, all initial seed inputs are kept in the corpus regardless of
whether they find new coverage or not. Further, these seed inputs are not
replaced with smaller inputs even if -reduce_inputs=1.

Differential Revision: https://reviews.llvm.org/D86577

show more ...

Reland "[NFC][libFuzzer] Prefix TempPath with string showing the work it is doing."

With fix (somehow one hunk is missed).

Revert "[NFC][libFuzzer] Prefix TempPath with string showing the work it is doing."

This reverts commit 8a29cb4421f1196bc40c4db5298ca13df516bf19.

fuzzer-linux bot has failure because of this.

[NFC][libFuzzer] Prefix TempPath with string showing the work it is doing.

[libFuzzer] implement a better queue for the fork mode. Add an internal flag -stop_file to allow graceful shutdown of fuzzing. Enhance the logging in the fork mode

llvm-svn: 363470

[libFuzzer] simplify the DFT trace collection using the new faster DFSan mode that traces up to 16 labels at a time and never runs out of labels. Second attempt. This time with a fix for windows (put

[libFuzzer] simplify the DFT trace collection using the new faster DFSan mode that traces up to 16 labels at a time and never runs out of labels. Second attempt. This time with a fix for windows (putenv instead of setenv))

llvm-svn: 363445

show more ...

Revert r363326 "[libFuzzer] simplify the DFT trace collection using the new faster DFSan mode that traces up to 16 labels at a time and never runs out of labels."

It broke the Windows build:

C:\b\s

Revert r363326 "[libFuzzer] simplify the DFT trace collection using the new faster DFSan mode that traces up to 16 labels at a time and never runs out of labels."

It broke the Windows build:

C:\b\s\w\ir\cache\builder\src\third_party\llvm\compiler-rt\lib\fuzzer\FuzzerDataFlowTrace.cpp(243): error C3861: 'setenv': identifier not found

This also reverts the follow-up r363327.

llvm-svn: 363358

show more ...

[libFuzzer] simplify the DFT trace collection using the new faster DFSan mode that traces up to 16 labels at a time and never runs out of labels.

llvm-svn: 363326

[libFuzzer] automatically collect the data flow trace (DFT) in the fork mode if -collect_data_flow= is given

llvm-svn: 361448

[libFuzzer] Replace -seed_corpus to better support fork mode on Win

Summary:
Pass seed corpus list in a file to get around argument length limits on Windows.
This limit was preventing many uses of f

[libFuzzer] Replace -seed_corpus to better support fork mode on Win

Summary:
Pass seed corpus list in a file to get around argument length limits on Windows.
This limit was preventing many uses of fork mode on Windows.

Reviewers: kcc, morehouse

Reviewed By: kcc

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D60980

llvm-svn: 359610

show more ...

[libFuzzer] in fork mode, make sure the child stats are always collected. This should fix the flakiness in fork-ubsan.test

llvm-svn: 358726

[libFuzzer] Remove tautological compare.

llvm-svn: 358510

[libFuzzer] speedup the merge step in the fork mode by merging only the files that have unique features.

llvm-svn: 358320

[libFuzzer] support -runs=N in the fork mode. Make sure we see one-line reports from ubsan in the fork mode. Test both

llvm-svn: 358306

[libFuzzer][Windows] Port fork mode to Windows

Summary:
Port libFuzzer's fork mode to Windows.
Implement Windows versions of MkDir, RmDir, and IterateDirRecursive to do this.
Don't print error messa

[libFuzzer][Windows] Port fork mode to Windows

Summary:
Port libFuzzer's fork mode to Windows.
Implement Windows versions of MkDir, RmDir, and IterateDirRecursive to do this.
Don't print error messages under new normal uses of FileSize (on a non-existent file).
Implement portable way of piping output to /dev/null.
Fix test for Windows and comment fork-sigusr.test on why it won't be ported to Win.

Reviewers: zturner

Reviewed By: zturner

Subscribers: kcc, zturner, jdoerfert, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D58513

llvm-svn: 355019

show more ...

[libFuzzer] fork mode: try harder to cleanup after itself

llvm-svn: 354186

[libFuzzer] form mode: add -ignore_crashes flag, honor the max_total_time flag, print the number of ooms/timeouts/crashes, fix a typo

llvm-svn: 354175

[libFuzzer] print new functions as they are discovered in the fork mode

llvm-svn: 354092