Fuzz Testing
What is Fuzz Testing?
In software testing, fuzzing is the practice of forcing applications to
consume corrupted data and observing the results. Poorly coded
applications will “fall over” when they get fuzzed data, typically
trying to process the data without checking to see if it’s correct and
complete. Well-coded applications will not crash, nor will they become
unstable because they reject improper data. The practice of fuzz testing
can quickly uncover dramatic coding errors and is relatively simple to
do, which explains its increasing popularity in the test community and
its use in the Microsoft® SDL. Unfortunately, the same things that make
it popular with testers, make it popular with hackers whose goal is to crash applications and look for sensitive data. This
popularity with hackers now makes the practice in the software
development lab a requirement, not just a cool new way to
test—applications have to be prepared for fuzzing.
How Does it Work?
Fuzz testing is accomplished by attacking all of an application’s data
interfaces, typically the file system, network, libraries, registry and
GUI. In its simplest form, fuzzing is accomplished using randomized
data—a jumbled raw network stream for example. This type of attack will
cause poorly coded applications to crash, but are non-deterministic in
nature and just tell the software team that there’s a problem and points
to its general locus. More sophisticated fuzzing includes parametric
data designed to attack specific parts of an application—a corrupted
word processing file header for example. This type of testing provides
deterministic results that allow software teams to zero in on specific
areas of concern and attack in a very focused and reproducible manner.
Both are good, both provide valuable data on the fragility and security
of an application.
Specialized tooling is not necessary to effectively
fuzz test an application. It’s a simple thing to make a
spreadsheet tool try to consume an image file and see what
happens. It’s also simple to break out a hex editor and change
values in a file header. However, it is a tedious process and
automation allows a greater range and depth of fuzz testing -
which is where Holodeck can assist. A dramatic example of where
random fuzz testing needs automation is in the Microsoft® SDL
where applications are required to consume 100,000 fuzzed files.
If an application fails to open just one, be it the first or the
99,999th, they have to be repaired and retested using the entire
file collection. Creating 100,000 files for testing is a big job
and automated tooling can help dramatically. A decent automated
tool will generate a collection of randomly fuzzed files with
the proper extensions for the application to consume; a better
tool will use a sample collection of files as a template and do
very specific fuzzing to specific parts of files; and a great
tool will virtualize the file stream and obviate the need for
physical files in the first place. In any case, the benefit of
automation in a large test file collection is as clear as the
benefits of fuzz testing itself.
Files are a single input point in many applications, though most
modern applications are less dependent on files, instead
consuming data across networks from servers of various types.
The reality is that fuzzing network streams is now as important,
if not more important than fuzzing file streams. Applying random
fuzzing techniques to network streams will clearly lead to
exposing some defects, but the really interesting stuff is in
parametric fuzzing where specific data are randomly changed or
replaced with specific data. Automated mechanisms for changing
specific parts data returned by a SQL command issued by the
client can have dramatic effects on application behavior, the
same as playing with the header in a word processing file.
Changes in table shape, changing the number of columns or
perhaps the row names may provide interesting results as might
corrupting the data in a specific field. In any case, fuzz
testing will help to uncover defects that would normally go
unfound, until of course the product ships and some user’s
network card breaks or a server starts spewing garbage for some
reason.
Holodeck and Fuzz Testing
Application vulnerabilities may be exposed using Holodeck, and before a hacker
discovers them in the wild. Tooling is available at various levels
from fairly good to great (where a great tool is one that allows
testers to virtualize data and parameterize data fed to
applications.) This is precisely what Holodeck does and
the basis for its creation.
Fuzz Test the Application’s I/O Vectors and Force it to
Consume Corrupted Data
Poorly coded applications will crash when they get fuzzed
[randomly corrupted] data, typically trying and process the data
without checking to see if it's correct and complete. For
example if a word processor just assumes that a .doc file is its
own and it turns out to be a binary image file, the word
processor will try and display the bits as a document and likely
crash. Fuzzing can be applied to everywhere an application gets
input data including file streams, network streams, GUI and the
registry. Holodeck gives testers the ability to quickly
virtualize the application under test’s runtime environment and
corrupt any or all of the application’s input vectors, forcing
it to consume bad data and crashing.
