Function-level obfuscation detection method based on Graph Convolutional Networks

Abstract The obfuscation detection technology is an important auxiliary means of malware detection. Also, for security practitioners, it can carry out automatic obfuscation detection before manual reverse analysis, which helps reverse engineers to perform reverse analysis more specifically. Existing obfuscation detection methods are mainly for Android applications and based on traditional machine learning, whose detection granularity is coarse, generality is poor, and the performance is not good enough. To address these issues, in this paper, we propose a function-level obfuscation detection method based on Graph Convolutional Networks for X86 assembly code and Android applications. Firstly, our method is function-level obfuscation detection, and we extract the Control Flow Graph (CFG) of each function as its feature, including the adjacency matrix and the basic block feature matrix. Secondly, we build a hybrid neural network model GCN-LSTM as our obfuscation detection model, which combines the Graph Convolutional Network (GCN) and the Long Short-Term Memory (LSTM). Finally, we conduct experiments using real-world open-source programs and compare results with baseline methods. For function-level detection, the accuracy of our method is 94.7575% for X86 assembly code and 98.9457% for Android applications, both of which are better than baseline methods. For APK-level detection, our method can almost completely detect the obfuscated APKs. Experimental results show that our method performs well for both X86 assembly code and Android applications and is superior to the baseline methods in both function-level detection and APK-level detection. Our research showcases a successful application of the Graph Convolutional Network and the Control Flow Graph on code obfuscation detection problems.