What is the fuss about Fuzzing?

We are all way past the time when we didn’t put a lot of effort into testing. We do it in numerous ways. Tests to see if the functions do what they are supposed to. Other tests to see if functionality is quick enough or can withstand the test of time. We often split it into functional and non-functional testing.

Here at Westermo where I currently work we test our software very thoroughly. We easily have too many tests written for various scenarios that we can not possibly run them all on a regular, i.e. nightly, basis. This led us to research (e.g. https://www.diva-portal.org/smash/get/diva2:1602576/FULLTEXT02.pdf) and develop a framework that picks the best tests to run for the night to keep on delivering software with high quality. This is of course on-going work.

However, one problem, in particular, is extra challenging: doing security testing. Why is that?

The challenge with testing for security is that it is not about the function doing the right thing or acting fast enough. It is about not doing the wrong thing, the negative. Why is that a problem?

It comes down to the number of possibilities. One easy example is input validation. See this classic example from xkcd, https://xkcd.com/327/.

Another example could be the old SYN flood attack (https://en.wikipedia.org/wiki/SYN_flood) which leads to a DoS. This one have a lot of countermeasures and should not be a problem for you today.

I could build a long list of similar attacks, scenarios, or bad implementations but I think the point is proven that we need to test for security. But how to do that? Enter the world of Fuzzing or Fuzz testing.

1. Automate! Since the number configuration combinations is almost endless and the innovation capabilities of the hacker community have no bounds this testing needs to be automated because you can never have enough testers employed.
2. Randomize! You need to to find a good distribution of the out of bounds and malformed things you test for. Whether it is a bad packet, bad input, or bad configuration, etc. the testing needs to be a good statistical distribution to cover as much as possible.
3. Threat model! This is not by definition testing but the result can serve as input to the other categories. Depending on your system and how it is built a threat model might point to say configuration being a weak spot, the automated and randomized process should reflect that and be weighted to counter the threat.

For me, these 3 simple steps of Fuzzing can in the long run increase your security. Another thing to consider and think about is how to measure the success of the testing. It will not be as clear as functional testing with pass/fail but more complicated. My suggestion is to tie the fuzz results (number of tests, pass ratio, random distribution, etc.) to the threat you are trying to mitigate. This can be communicated to customers as a way of building trust in your security posture.