Fuzzing with multiple servers in parallel: AFL++ with Network File Systems

Introduction ##

When fuzzing large-scale applications, using a single server (even with 4 64-core AMD Ryzen CPUs) may not be powerful enough by itself. That’s where parallelized/distributed fuzzing comes in (i.e. automatic sharing of results between fuzzing systems). In this guide, we’ll take a look at how to set up multiple servers fuzzing the same program using AFL++, linked all together with an NFS (Network File System).

Instructions ##

Step 1: Set up the NFS servers ###

To start, we need to set up an NFS on on each of the systems we’re going to be fuzzing on. In this post, we’re going to use four servers named fuzz, buzz, ping, and pong. Each server has the IP addresses 10.0.0.10, 10.0.0.11, 10.0.0.12, and 10.0.0.13, respectively.

We’ll start with fuzz.

1. Install NFS server, create a directory (which we will be exporting), and set the permissions:

sudo apt install nfs-kernel-server
sudo mkdir -p /mnt/fuzz/
sudo chown -R nobody:nogroup /mnt/buzz
sudo chmod -R 666 /mnt/fuzz

2. Edit the NFS export configuration. In this configuration, we limit the access to /mnt/fuzz to the addresses of the four systems:

sudo su
cat << EOF >> /etc/exports
/mnt/fuzz  10.0.0.10/32(rw,insecure,async,no_root_squash,no_subtree_check)
/mnt/fuzz  10.0.0.11/32(rw,insecure,async,no_root_squash,no_subtree_check)
/mnt/fuzz  10.0.0.12/32(rw,insecure,async,no_root_squash,no_subtree_check)
/mnt/fuzz  10.0.0.13/32(rw,insecure,async,no_root_squash,no_subtree_check)
EOF
exit

3. Export the configuration and restart the server:

sudo exportfs -a
sudo systemctl restart nfs-kernel-server

We then need to repeat this on each of the other servers. To set up the buzz server, we follow the instructions exactly the same, but replacing any reference to fuzz with buzz.

Step 2: Mount the remote NFS servers ###

Now that we’ve set up the NFS servers, we now need to mount each of the remote filesystems on each server.

sudo apt install nfs-common

[ -d "/mnt/fuzz" ] || (sudo mkdir /mnt/fuzz && sudo mount -o r,noacl,nocto,nodiratime,noatime,bg,rsize=32768,wsize=32768 10.0.0.10:/mnt/fuzz /mnt/fuzz)

[ -d "/mnt/buzz" ] || (sudo mkdir /mnt/buzz && sudo mount -o r,noacl,nocto,nodiratime,noatime,bg,rsize=32768,wsize=32768 10.0.0.11:/mnt/buzz /mnt/buzz)

[ -d "/mnt/ping" ] || (sudo mkdir /mnt/ping && sudo mount -o r,noacl,nocto,nodiratime,noatime,bg,rsize=32768,wsize=32768 10.0.0.12:/mnt/ping /mnt/ping)

[ -d "/mnt/pong" ] || (sudo mkdir /mnt/pong && sudo mount -o r,noacl,nocto,nodiratime,noatime,bg,rsize=32768,wsize=32768 10.0.0.13:/mnt/pong /mnt/pong)

Basically, we mount all of the remote NFS’ on each server: if the directory already exists, then there’s no need to mount it since it’s the local version. If a directory is ever unmounted (so, during a reboot), then none of the mounts will run; you should delete the empty directories.

Step 3: Start fuzzing on the servers ###

Now we set up the fuzzing on each server. In this case, we start on fuzz.

When fuzzing, we are outputting the local crashes to /mnt/fuzz/ and treat all of the remote NFS’ as “foreign fuzzers”: read-only. Locally, all fuzzers (using the -M and -S flags) already share results, so we only need to specify the main queue for each foreign fuzzer: The flags are as followed:

afl-fuzz -i corpus/ -M fuzz_1 -o /mnt/fuzz/ -F /mnt/buzz/buzz_1/queue/ -F /mnt/ping/ping_1/queue/ -F /mnt/pong/pong_1/queue/ ./fuzzed-program

Still on the fuzz server, we then fuzz with all of the cores available:

afl-fuzz -i corpus/ -S fuzz_2 -o /mnt/fuzz/ ./fuzzed-program
afl-fuzz -i corpus/ -S fuzz_3 -o /mnt/fuzz/ ./fuzzed-program
...

All of the flags used are as followed:

  -i dir        - input directory with test cases
  -o dir        - output directory for fuzzer findings
  -M/-S id      - distributed mode (-M sets -Z and disables trimming)
  -F path       - sync to a foreign fuzzer queue directory (requires -M)

Basically, afl-fuzz on this machine is continuously reading and writing to the local /mnt/fuzz, and periodically checking the remote /mnt/buzz/buzz_1/queue/, /mnt/ping/ping_1/queue/, and /mnt/pong/pong_1/queue/ for any new findings from the other servers. Checking those remote directories can be slow due to the network overhead, however since they are not continuously reading the remote locations, the overhead is limited.

Finally, we repeat this process on all of the other servers.

On the buzz server, we would run:

afl-fuzz -i corpus/ -M buzz_1 -o /mnt/buzz/ -F /mnt/fuzz/fuzz_1/queue/ -F /mnt/ping/ping_1/queue/ -F /mnt/pong/pong_1/queue/ ./fuzzed-program

and

afl-fuzz -i corpus/ -S buzz_2 -o /mnt/buzz/ ./fuzzed-program
afl-fuzz -i corpus/ -S buzz_3 -o /mnt/buzz/ ./fuzzed-program
...

And similarly on the other servers.

Conclusion ##

Syncing data between multiple systems in order to fuzz a target in parallel is a problem that can be solved in multiple ways. By using NFS’ to make the queue directories of other systems available, we create a fairly easy-to-scale solution which fits the job as needed. What’s more, a simple bash script can enumerate all of the instructions in this blog post quite easily, assuming conformity in the hostnames of each of the systems used for fuzzing: but that’s for another time…