ReFSM: Reverse engineering from protocol packet traces to test generation by extended finite state machines

Abstract Protocol reverse engineering is helpful to automatically obtain the specifications of protocols that are useful for network management, network security systems and test case generation tools. To achieve better accuracy, these kinds of applications require good models that can capture not only the order of exchanging messages (control flow aspect) but also the data being transmitted (data flow aspect). However, current techniques only focus on inferring the control flow represented as a Finite State Machine (FSM) and without interpreting the data flow. The Extended Finite State Machine (EFSM), embedding memory in the states and data guard in the FSM transitions, is a method commonly used to represent the data flow. In this work, we propose ReFSM, a novel approach to infer the EFSMs of protocols from only network packet traces. The proposed method is evaluated by using datasets of real-world network traffic traces of four protocols: FTP, SMTP, BitTorrent and PPLive. Based on the results, the coverage, accuracy scores of correctness and behavior of inferred models are always higher than 90%. The precision and recall values of message type identification are, at least, well above 94% and 96%, respectively. The inferred EFSMs are close to the correct model derived from protocol specification.