STATIC FEATURE-BASED MALWARE CLASSIFICATION

Abstract

Malware poses a serious threat to modern computer systems and user information, causing significant economic losses and security risks. Therefore, developing effective and scalable malware detection methods remains a critical challenge. Among existing approaches, static analysis based on Portable Executable (PE) file features is widely adopted due to its safety, efficiency, and suitability for large-scale deployment. This thesis proposes a static malware classification system utilizing tree-based machine learning models, including LightGBM, XGBoost, CatBoost, and Random Forest. Experiments are conducted on three datasets with distinct static characteristics: EMBER2018, MalwareBazaar, and a multi-class custom dataset. Model performance is evaluated using standard metrics such as Accuracy, Precision, Recall, F1-score, ROC-AUC, and inference time. In addition to in-domain evaluation, cross-domain experiments are conducted to assess the models' generalization under domain-shift conditions. The experimental results show that while the models achieve high accuracy and low inference latency in in-domain scenarios, their performance degrades significantly in cross-dataset evaluations, particularly when feature distributions or class structures differ across datasets. These findings highlight the substantial impact of domain shift and underscore the need for additional training strategies, fine-tuning, or domain adaptation techniques to ensure robust malware detection in real-world deployments. Keywords: Malware detection, Static analysis, Portable Executable, Machine learning, Domain shift, Cross-dataset evaluation.