Fuzz Target#

The first step in using libFuzzer on a library is to implement a fuzz target – a function that accepts an array of bytes and does something interesting with these bytes using the API under test. Like this:

// fuzz_target.cc
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  DoSomethingInterestingWithMyAPI(Data, Size);
  return 0;  // Values other than 0 and -1 are reserved for future use.
}

Note that this fuzz target does not depend on libFuzzer in any way and so it is possible and even desirable to use it with other fuzzing engines e.g. AFL and/or Radamsa.

Some important things to remember about fuzz targets:

• The fuzzing engine will execute the fuzz target many times with different inputs in the same process.

• It must tolerate any kind of input (empty, huge, malformed, etc).

• It must not exit() on any input.

• It may use threads but ideally all threads should be joined at the end of the function.

• It must be as deterministic as possible. Non-determinism (e.g. random decisions not based on the input bytes) will make fuzzing inefficient.

• It must be fast. Try avoiding cubic or greater complexity, logging, or excessive memory consumption.

• Ideally, it should not modify any global state (although that’s not strict).

• Usually, the narrower the target the better. E.g. if your target can parse several data formats, split it into several targets, one per format.

Fuzzer Usage#

Recent versions of Clang (starting from 6.0) include libFuzzer, and no extra installation is necessary.

In order to build your fuzzer binary, use the -fsanitize=fuzzer flag during the compilation and linking. In most cases you may want to combine libFuzzer with AddressSanitizer (ASAN), UndefinedBehaviorSanitizer (UBSAN), or both. You can also build with MemorySanitizer (MSAN), but support is experimental:

 clang -g -O1 -fsanitize=fuzzer                         mytarget.c # Builds the fuzz target w/o sanitizers
 clang -g -O1 -fsanitize=fuzzer,address                 mytarget.c # Builds the fuzz target with ASAN
 clang -g -O1 -fsanitize=fuzzer,signed-integer-overflow mytarget.c # Builds the fuzz target with a part of UBSAN
 clang -g -O1 -fsanitize=fuzzer,memory                  mytarget.c # Builds the fuzz target with MSAN

This will perform the necessary instrumentation, as well as linking with the libFuzzer library. Note that -fsanitize=fuzzer links in the libFuzzer’s main() symbol.

If modifying CFLAGS of a large project, which also compiles executables requiring their own main symbol, it may be desirable to request just the instrumentation without linking:

 clang -fsanitize=fuzzer-no-link mytarget.c

Then libFuzzer can be linked to the desired driver by passing in -fsanitize=fuzzer during the linking stage.

Corpus#

Coverage-guided fuzzers like libFuzzer rely on a corpus of sample inputs for the code under test. This corpus should ideally be seeded with a varied collection of valid and invalid inputs for the code under test; for example, for a graphics library the initial corpus might hold a variety of different small PNG/JPG/GIF files. The fuzzer generates random mutations based around the sample inputs in the current corpus. If a mutation triggers execution of a previously-uncovered path in the code under test, then that mutation is saved to the corpus for future variations.

LibFuzzer will work without any initial seeds, but will be less efficient if the library under test accepts complex, structured inputs.

The corpus can also act as a sanity/regression check, to confirm that the fuzzing entrypoint still works and that all of the sample inputs run through the code under test without problems.

If you have a large corpus (either generated by fuzzing or acquired by other means) you may want to minimize it while still preserving the full coverage. One way to do that is to use the -merge=1 flag:

mkdir NEW_CORPUS_DIR  # Store minimized corpus here.
./my_fuzzer -merge=1 NEW_CORPUS_DIR FULL_CORPUS_DIR

You may use the same flag to add more interesting items to an existing corpus. Only the inputs that trigger new coverage will be added to the first corpus.

./my_fuzzer -merge=1 CURRENT_CORPUS_DIR NEW_POTENTIALLY_INTERESTING_INPUTS_DIR

Running#

To run the fuzzer, first create a Corpus directory that holds the initial “seed” sample inputs:

mkdir CORPUS_DIR
cp /some/input/samples/* CORPUS_DIR

Then run the fuzzer on the corpus directory:

./my_fuzzer CORPUS_DIR  # -max_len=1000 -jobs=20 ...

As the fuzzer discovers new interesting test cases (i.e. test cases that trigger coverage of new paths through the code under test), those test cases will be added to the corpus directory.

By default, the fuzzing process will continue indefinitely – at least until a bug is found. Any crashes or sanitizer failures will be reported as usual, stopping the fuzzing process, and the particular input that triggered the bug will be written to disk (typically as crash-<sha1>, leak-<sha1>, or timeout-<sha1>).

Parallel Fuzzing#

Each libFuzzer process is single-threaded, unless the library under test starts its own threads. However, it is possible to run multiple libFuzzer processes in parallel with a shared corpus directory; this has the advantage that any new inputs found by one fuzzer process will be available to the other fuzzer processes (unless you disable this with the -reload=0 option).

This is primarily controlled by the -jobs=N option, which indicates that that N fuzzing jobs should be run to completion (i.e. until a bug is found or time/iteration limits are reached). These jobs will be run across a set of worker processes, by default using half of the available CPU cores; the count of worker processes can be overridden by the -workers=N option. For example, running with -jobs=30 on a 12-core machine would run 6 workers by default, with each worker averaging 5 bugs by completion of the entire process.

Fork mode#

Experimental mode -fork=N (where N is the number of parallel jobs) enables oom-, timeout-, and crash-resistant fuzzing with separate processes (using fork-exec, not just fork).

The top libFuzzer process will not do any fuzzing itself, but will spawn up to N concurrent child processes providing them small random subsets of the corpus. After a child exits, the top process merges the corpus generated by the child back to the main corpus.

Related flags:

-ignore_ooms

True by default. If an OOM happens during fuzzing in one of the child processes, the reproducer is saved on disk, and fuzzing continues.

-ignore_timeouts

True by default, same as -ignore_ooms, but for timeouts.

-ignore_crashes

False by default, same as -ignore_ooms, but for all other crashes.

The plan is to eventually replace -jobs=N and -workers=N with -fork=N.

Resuming merge#

Merging large corpora may be time consuming, and it is often desirable to do it on preemptable VMs, where the process may be killed at any time. In order to seamlessly resume the merge, use the -merge_control_file flag and use killall -SIGUSR1 /path/to/fuzzer/binary to stop the merge gracefully. Example:

% rm -f SomeLocalPath
% ./my_fuzzer CORPUS1 CORPUS2 -merge=1 -merge_control_file=SomeLocalPath
...
MERGE-INNER: using the control file 'SomeLocalPath'
...
# While this is running, do `killall -SIGUSR1 my_fuzzer` in another console
==9015== INFO: libFuzzer: exiting as requested

# This will leave the file SomeLocalPath with the partial state of the merge.
# Now, you can continue the merge by executing the same command. The merge
# will continue from where it has been interrupted.
% ./my_fuzzer CORPUS1 CORPUS2 -merge=1 -merge_control_file=SomeLocalPath
...
MERGE-OUTER: non-empty control file provided: 'SomeLocalPath'
MERGE-OUTER: control file ok, 32 files total, first not processed file 20
...

Toy example#

A simple function that does something interesting if it receives the input “HI!”:

cat << EOF > test_fuzzer.cc
#include <stdint.h>
#include <stddef.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
  if (size > 0 && data[0] == 'H')
    if (size > 1 && data[1] == 'I')
       if (size > 2 && data[2] == '!')
       __builtin_trap();
  return 0;
}
EOF
# Build test_fuzzer.cc with asan and link against libFuzzer.
clang++ -fsanitize=address,fuzzer test_fuzzer.cc
# Run the fuzzer with no corpus.
./a.out

You should get an error pretty quickly:

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1434179311
INFO: Loaded 1 modules   (8 inline 8-bit counters): 8 [0x5f03d189be90, 0x5f03d189be98),
INFO: Loaded 1 PC tables (8 PCs): 8 [0x5f03d189be98,0x5f03d189bf18),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 2 ft: 2 corp: 1/1B exec/s: 0 rss: 31MB
#144    NEW    cov: 3 ft: 3 corp: 2/2B lim: 4 exec/s: 0 rss: 31MB L: 1/1 MS: 2 ChangeByte-ChangeByte-
#157    NEW    cov: 4 ft: 4 corp: 3/4B lim: 4 exec/s: 0 rss: 31MB L: 2/2 MS: 3 CrossOver-ChangeBit-CrossOver-
#1345   NEW    cov: 5 ft: 5 corp: 4/8B lim: 14 exec/s: 0 rss: 32MB L: 4/4 MS: 3 InsertByte-ChangeBit-CrossOver-
#1696   NEW    cov: 6 ft: 6 corp: 5/10B lim: 17 exec/s: 0 rss: 32MB L: 2/4 MS: 1 EraseBytes-
#1832   REDUCE cov: 6 ft: 6 corp: 5/9B lim: 17 exec/s: 0 rss: 32MB L: 3/3 MS: 1 EraseBytes-
==840148== ERROR: libFuzzer: deadly signal
...
SUMMARY: libFuzzer: deadly signal
MS: 2 CopyPart-ChangeByte-; base unit: dbee5f8c7a5da845446e75b4a5708e74428b520a
0x48,0x49,0x21,
HI!
artifact_prefix='./'; Test unit written to ./crash-7a8dc3985d2a90fb6e62e94910fc11d31949c348
Base64: SEkh

More examples#

Examples of real-life fuzz targets and the bugs they find can be found at http://tutorial.libfuzzer.info. Among other things you can learn how to detect Heartbleed in one second.

Dictionaries#

LibFuzzer supports user-supplied dictionaries with input language keywords or other interesting byte sequences (e.g. multi-byte magic values). Use -dict=DICTIONARY_FILE. For some input languages using a dictionary may significantly improve the search speed. The dictionary syntax is similar to that used by AFL for its -x option:

# Lines starting with '#' and empty lines are ignored.

# Adds "blah" (w/o quotes) to the dictionary.
kw1="blah"
# Use \\ for backslash and \" for quotes.
kw2="\"ac\\dc\""
# Use \xAB for hex values
kw3="\xF7\xF8"
# the name of the keyword followed by '=' may be omitted:
"foo\x0Abar"

Tracing CMP instructions#

With an additional compiler flag -fsanitize-coverage=trace-cmp (on by default as part of -fsanitize=fuzzer, see SanitizerCoverageTraceDataFlow) libFuzzer will intercept CMP instructions and guide mutations based on the arguments of intercepted CMP instructions. This may slow down the fuzzing but is very likely to improve the results.

Value Profile#

With -fsanitize-coverage=trace-cmp (default with -fsanitize=fuzzer) and extra run-time flag -use_value_profile=1 the fuzzer will collect value profiles for the parameters of compare instructions and treat some new values as new coverage.

The current implementation does roughly the following:

• The compiler instruments all CMP instructions with a callback that receives both CMP arguments.

• The callback computes (caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12) and uses this value to set a bit in a bitset.

• Every new observed bit in the bitset is treated as new coverage.

This feature has a potential to discover many interesting inputs, but there are two downsides. First, the extra instrumentation may bring up to 2x additional slowdown. Second, the corpus may grow by several times.

Fuzzer-friendly build mode#

Sometimes the code under test is not fuzzing-friendly. Examples:

• The target code uses a PRNG seeded e.g. by system time and thus two consequent invocations may potentially execute different code paths even if the end result will be the same. This will cause a fuzzer to treat two similar inputs as significantly different and it will blow up the test corpus. E.g. libxml uses rand() inside its hash table.

• The target code uses checksums to protect from invalid inputs. E.g. png checks CRC for every chunk.

In many cases it makes sense to build a special fuzzing-friendly build with certain fuzzing-unfriendly features disabled. We propose to use a common build macro for all such cases for consistency: FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.

void MyInitPRNG() {
#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
  // In fuzzing mode the behavior of the code should be deterministic.
  srand(0);
#else
  srand(time(0));
#endif
}

AFL compatibility#

LibFuzzer can be used together with AFL on the same test corpus. Both fuzzers expect the test corpus to reside in a directory, one file per input. You can run both fuzzers on the same corpus, one after another:

./afl-fuzz -i testcase_dir -o findings_dir /path/to/program @@
./llvm-fuzz testcase_dir findings_dir  # Will write new tests to testcase_dir

Periodically restart both fuzzers so that they can use each other’s findings. Currently, there is no simple way to run both fuzzing engines in parallel while sharing the same corpus dir.

You may also use AFL on your target function LLVMFuzzerTestOneInput: see an example here.

How good is my fuzzer?#

Once you implement your target function LLVMFuzzerTestOneInput and fuzz it to death, you will want to know whether the function or the corpus can be improved further. One easy to use metric is, of course, code coverage.

We recommend to use Clang Coverage, to visualize and study your code coverage (example).

User-supplied mutators#

LibFuzzer allows to use custom (user-supplied) mutators, see Structure-Aware Fuzzing for more details.

Startup initialization#

If the library being tested needs to be initialized, there are several options.

The simplest way is to have a statically initialized global object inside LLVMFuzzerTestOneInput (or in global scope if that works for you):

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  static bool Initialized = DoInitialization();
  ...

Alternatively, you may define an optional init function and it will receive the program arguments that you can read and modify. Do this only if you really need to access argv/argc.

 extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv) {
  ReadAndMaybeModify(argc, argv);
  return 0;
 }

Using libFuzzer as a library#

If the code being fuzzed must provide its own main, it’s possible to invoke libFuzzer as a library. Be sure to pass -fsanitize=fuzzer-no-link during compilation, and link your binary against the no-main version of libFuzzer. On Linux installations, this is typically located at:

/usr/lib/<llvm-version>/lib/clang/<clang-version>/lib/linux/libclang_rt.fuzzer_no_main-<architecture>.a

If building libFuzzer from source, this is located at the following path in the build output directory:

lib/linux/libclang_rt.fuzzer_no_main-<architecture>.a

From here, the code can do whatever setup it requires, and when it’s ready to start fuzzing, it can call LLVMFuzzerRunDriver, passing in the program arguments and a callback. This callback is invoked just like LLVMFuzzerTestOneInput, and has the same signature.

extern "C" int LLVMFuzzerRunDriver(int *argc, char ***argv,
                  int (*UserCb)(const uint8_t *Data, size_t Size));

Rejecting unwanted inputs#

It may be desirable to reject some inputs, i.e. to not add them to the corpus.

For example, when fuzzing an API consisting of parsing and other logic, one may want to allow only those inputs into the corpus that parse successfully.

If the fuzz target returns -1 on a given input, libFuzzer will not add that input top the corpus, regardless of what coverage it triggers.

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
  if (auto *Obj = ParseMe(Data, Size)) {
    Obj->DoSomethingInteresting();
    return 0;  // Accept. The input may be added to the corpus.
  }
  return -1;  // Reject; The input will not be added to the corpus.
}

Leaks#

Binaries built with AddressSanitizer or LeakSanitizer will try to detect memory leaks at the process shutdown. For in-process fuzzing this is inconvenient since the fuzzer needs to report a leak with a reproducer as soon as the leaky mutation is found. However, running full leak detection after every mutation is expensive.

By default (-detect_leaks=1) libFuzzer will count the number of malloc and free calls when executing every mutation. If the numbers don’t match (which by itself doesn’t mean there is a leak) libFuzzer will re-execute the input to verify the mismatch and then invoke the more expensive LeakSanitizer pass and if the actual leak is found, it will be reported with the reproducer and the process will exit. The verification re-executions are not fuzzing runs and do not count towards -runs.

If your target has massive leaks and the leak detection is disabled you will eventually run out of RAM (see the -rss_limit_mb flag).

Q. Why doesn’t libFuzzer use any of the LLVM support?#

There are two reasons.

First, we want this library to be used outside of the LLVM without users having to build the rest of LLVM. This may sound unconvincing for many LLVM folks, but in practice the need for building the whole LLVM frightens many potential users – and we want more users to use this code.

Second, there is a subtle technical reason not to rely on the rest of LLVM, or any other large body of code (maybe not even STL). When coverage instrumentation is enabled, it will also instrument the LLVM support code which will blow up the coverage set of the process (since the fuzzer is in-process). In other words, by using more external dependencies we will slow down the fuzzer while the main reason for it to exist is extreme speed.

Q. Does libFuzzer Support Windows?#

Yes, libFuzzer now supports Windows. Initial support was added in r341082. Any build of Clang 9 supports it. You can download a build of Clang for Windows that has libFuzzer from LLVM Snapshot Builds.

Using libFuzzer on Windows without ASAN is unsupported. Building fuzzers with the /MD (dynamic runtime library) compile option is unsupported. Support for these may be added in the future. Linking fuzzers with the /INCREMENTAL link option (or the /DEBUG option which implies it) is also unsupported.

Send any questions or comments to the mailing list: libfuzzer(#)googlegroups.com

Q. When libFuzzer is not a good solution for a problem?#

• If the test inputs are validated by the target library and the validator asserts/crashes on invalid inputs, in-process fuzzing is not applicable.

• Bugs in the target library may accumulate without being detected. E.g. a memory corruption that goes undetected at first and then leads to a crash while testing another input. This is why it is highly recommended to run this in-process fuzzer with all sanitizers to detect most bugs on the spot.

• It is harder to protect the in-process fuzzer from excessive memory consumption and infinite loops in the target library (still possible).

• The target library should not have significant global state that is not reset between the runs.

• Many interesting target libraries are not designed in a way that supports the in-process fuzzer interface (e.g. require a file path instead of a byte array).

• If a single test run takes a considerable fraction of a second (or more) the speed benefit from the in-process fuzzer is negligible.

• If the target library runs persistent threads (that outlive execution of one test) the fuzzing results will be unreliable.

Q. So, what exactly this Fuzzer is good for?#

This Fuzzer might be a good choice for testing libraries that have relatively small inputs, each input takes < 10ms to run, and the library code is not expected to crash on invalid inputs. Examples: regular expression matchers, text or binary format parsers, compression, network, crypto.

Q. LibFuzzer crashes on my complicated fuzz target (but works fine for me on smaller targets).#

Check if your fuzz target uses dlclose. Currently, libFuzzer doesn’t support targets that call dlclose, this may be fixed in future.