Syntax-aware differential fuzzing of black-box network protocol implementations

Abstract

Protocols for secure network communication are a cornerstone of our digitalized life, but implementing them is a complicated and error-prone task. Most such protocols are very complex, while their specifications tend to come as voluminous and sometimes ambiguous prose text. As a result, flaws in their implementations are hard to avoid, but easily cause interoperability issues or impair security. Identifying flaws in (i.e., testing) such implementations is of highest importance. One technique that has proven very successful for implementation testing is fuzzing. One of its key ideas is to generate test input in a randomized way. If the implementation under test runs into some exceptional execution state (e.g., crashes) while processing the input, a flaw in the tested implementation has been found. The outstanding advantage of fuzzing is that it allows full automation and features a very low barrier to deployment. On the other hand, its disadvantage is that relying on exceptional execution states typically implies the need for invasive forms of program monitoring and defeats the detectability of semantic flaws. Moreover is unspecific randomized input generation hardly suitable for testing implementations that expect complex and highly-structured input data. When it comes to testing implementations of complex network protocols like such for secure communication, however, both semantic flaws and complex message formats are a key concern. Therefore, in this thesis, we propose and study a novel concept called syntax-aware differential protocol fuzzing (SDPF). It addresses the aforementioned disadvantages of fuzzing in the following way. First, SDPF includes a novel protocol-independent, fuzzing-based, and response-guided input generation approach. It can be customized to become syntax-aware for binary-encoded messages with little manual effort. This allows to efficiently generate mostlyvalid protocol messages even for very complex message structures. Second, SDPF makes use of differential testing, i.e., it uses multiple implementations of the same protocol to gather evidence of implementation flaws from behavioural discrepancies. This can spare invasive program monitoring to some extent and enable sensitivity to semantic flaws without the need to explicitly distinguish correct from potentially incorrect implementation behaviour. Our work was the first to make differential testing applicable to the message processing and protocol logic routines of complex network protocols. Using the Transport Layer Security (TLS) protocol as a representative example, we experimentally show that our input generation approach finds more behavioural discrepancies and achieves better code coverage than other approaches. As is the case for differential testing in general, tracking down the root cause of a behavioural discrepancies remains as a manual task for a human tester. Unfortunately, the complexity of test messages often makes such manual investigation very timeconsuming. Besides, optional protocol features and liberal specifications can cause false positives and give rise to unprofitable investigation effort. To address this, we propose and study a novel concept called message feature-based differential testing. It probes and contrasts how different implementations react to the presence or absence of well-defined properties in stimulation messages. One of the advantages of message feature-based differential testing over classical differential testing is that, for each discrepancy found, the respective message property naturally provides a very specific hint for a human tester where to start a manual investigation. Using TLS and its implementations as an example, we experimentally show how message featurebased differential testing can help to identify implementation flaws. As one of the key ingredients of SDPF and message feature-based differential testing, we propose and discuss a novel concept called Generic Message Trees (GMTs). A GMT is a versatile, tree-like data structure that enables a syntax-aware, user-friendly, unified, and efficient handling of binary-encoded network protocol messages. GMTs are one of the main drivers of the low-effort syntax awareness provided by our approaches. In the course of our experimental evaluations, we identified several previously-unknown implementation flaws in widespread open-source TLS implementations. Among them is a combination of two flaws in a single implementation, which in conjunction lead to a significant security vulnerability. Our work can be the basis for interesting further research in various directions. This includes, for example, the application to and evaluation on other protocols than TLS and the usability of our approach for remote implementation fingerprinting.