A3CMal: Generating adversarial samples to force targeted misclassification by reinforcement learning

Abstract Machine learning algorithms have been proved to be vulnerable to adversarial attacks. The potential adversary is able to force the model to produce deliberate errors by elaborately modifying the training samples. For malware analysis, most of the existing research on evasion attacks focuses on a detection scenario, while less attention is paid to the classification scenario which is vital to decide a suitable system response in time. To fulfill this gap, this paper tries to address the misclassification problem in malware analysis. A reinforcement learning model named A3CMal is proposed. This adversarial model aims to generate adversarial samples which can fool the target classifier. As a core component of A3CMal, the self-learning agent constantly takes optimal actions to confuse the classification by slightly modifying samples on the basis of the observed states. Extensive experiments are performed to test the validity of A3CMal. The results show that the proposed A3CMal can force the target classifier to make wrong predictions while preserving the malicious functionality of the malware. Remarkably, not only can it cause the system to indicate an incorrect classification, but also can mislead the target model to classify malware into a specific category. Furthermore, our experiments demonstrate that the PE-based classifier is vulnerable to the adversarial samples generated by A3CMal.