Bypassing anti-emulation methods for malware detection

Abstract

Malware detection has played a crucial role in many cyberattacks in recent years. Due to the obfuscated nature of malware, the traditional static analysis technique tends to be ineffective. Additionally, modern malware often can identify dynamic analysis environments, posing challenges to dynamic analysis methods. Thus, feature extraction relies on analysis techniques that tend to be less effective in obfuscated malware, resulting in poor performance of subsequent machine learning-based detectors. This study introduces a Bypass Anti-emulation-based Malware Detection framework (BAE-MD) for enhancing the efficiency of obfuscated malware detection. In other words, BAE-MD includes a method that can bypass the anti-emulation mechanism of malware in a controlled dynamic environment. This forces the malware to decrypt and decompress its actual malicious code to memory. By doing so, Yara rules can be applied to memory dump to extract more than $60$ features to feed into detectors. BAE-MD is evaluated on a malware dataset in comparison with others using static and dynamic analysis technique-based feature extraction. The experimental results can confirm that our method outperforms the others. More investigations are also carried out to illustrate the efficiency of BAE-MD. These results suggest that BAE-MD is a promising approach for dealing with the continuous evolution of malware.