A Multi-Perspective Malware Detection Approach Through Behavioral Fusion of API Call Sequence

Abstract The widespread development of the malware industry is considered the main threat to our e-society. Therefore, malware analysis should also be enriched with smart heuristic tools that recognize malicious behaviors effectively. Although the generated API calling graph representation for malicious processes encodes worthwhile information about their malicious behavior, it is pragmatically inconvenient to generate a behavior graph for each process. Therefore, we experimented with creating generic behavioral graph models that describe malicious and non-malicious processes. These behavioral models relied on the fusion of statistical, contextual, and graph mining features that capture explicit and implicit relationships between API functions in the calling sequence. Our generated behavioral models proved the behavioral contrast between malicious and non-malicious calling sequences. According to that distinction, we built different relational perspective models that characterize processes’ behaviors. To prove our approach novelty, we experimented with our approach over Windows and Android platforms. Our experimentations demonstrated that our proposed system identified unseen malicious samples with high accuracy with low false-positive. In terms of detection accuracy, our model returns an average accuracy of 0.997 and 0.977 to the unseen Windows and Android malware testing samples, respectively. Moreover, we proposed a new indexing method for APIs based on their contextual similarities. We also suggested a new expressive, a visualized form that renders the API calling sequence. Consequently, we introduced a confidence metric to our model classification decision. Furthermore, we developed a behavioral heuristic that effectively identified malicious API call sequences that were deceptive or mimicry.