A Deep Learning–Based Hybrid Neural Network Model for Malware Detection

Muthana S. Mahdi

doi:10.51173/jt.v8i1.2748

Authors

Muthana S. Mahdi Department of Computer Science, College of Science, Mustansiriyah University, Baghdad, Iraq

DOI:

https://doi.org/10.51173/jt.v8i1.2748

Keywords:

Malware Detection, Cybersecurity, Malware Classification, Residual Connections, Attention Mechanism, Recurrent Neural Networks (RNNs)

Abstract

Malware remains a significant threat to modern computing systems and networks worldwide. Evolving malware utilises polymorphism, metamorphism, and also zero-day exploits to bypass defenses. Traditional signature-based and heuristic detection methods are now struggling with the increasing complexity of malware. In this paper, a hybrid neural network model is proposed that combines a convolutional neural network (CNN) to detect spatial malware patterns, recurrent neural networks (RNNs) to analyses temporal behaviors, and also attention mechanisms to select crucial features for accurate and reliable threat classification. Multi-scale convolutional layers and residual connections improve dataset generalization and reduce overfitting. Focal loss functionality addresses the class imbalance in real-world malware detection scenarios. Experimental results on EMBER, EMBER Sim, and SoReL-20M datasets show exceptional accuracy and precision. This interpretable, scalable deep learning (DL) model bridges traditional methods with modern cybersecurity challenges. The model excels in zero-day detection and produces a few false positives, achieving 96.7% accuracy, 96.1% precision, 96.8% recall, and 96.4% F1-score. Additionally, the findings demonstrate clear improvements over previous methods, achieving a 1.1–2.6% increase in accuracy, confirming the model’s superior detection capability. This advanced deep-learning approach sets a new benchmark in cybersecurity.

Downloads

Download data is not yet available.

Author Biography

Muthana S. Mahdi, Department of Computer Science, College of Science, Mustansiriyah University, Baghdad, Iraq

References

J. Jeon, J. H. Park and Y. S. Jeong, “Dynamic Analysis for IoT Malware Detection with Convolution Neural Network Model,” IEEE Access, vol. 8, pp. 96899–96911, 2020, doi.org/10.1109/ACCESS.2020.2995887.

M. Woźniak, J. Siłka, M. Wieczorek and M. Alrashoud, “Recurrent Neural Network Model for IoT and Networking Malware Threat Detection,” IEEE Trans. Ind. Inform., vol. 17, pp. 5583–5594, 2020, doi.org/10.1109/TII.2020.3021689.

P. Yadav, N. Menon, V. Ravi, S. Vishvanathan and T. D. Pham, “EfficientNet Convolutional Neural Networks-Based Android Malware Detection,” Computers & Security, vol. 115, p. 102622, 2022, doi.org/10.1016/j.cose.2022.102622.

S. Jha, D. Prashar, H. V. Long and D. Taniar, “Recurrent Neural Network for Detecting Malware,” Computers & Security, vol. 99, p. 102037, 2020, doi.org/10.1016/j.cose.2020.102037.

Y. Jian, H. Kuang, C. Ren, Z. Ma and H. Wang, “A Novel Framework for Image-Based Malware Detection with a Deep Neural Network,” Computers & Security, vol. 109, p. 102400, 2021, doi.org/10.1016/j.cose.2021.102400.

J. Qiu, J. Zhang, W. Luo, L. Pan, S. Nepal and Y. Xiang, “A Survey of Android Malware Detection with Deep Neural Models,” ACM Comput. Surv., vol. 53, pp. 1–36, 2020, doi.org/10.1145/3417978.

M. S. Mahdi and Z. L. Ali, “A Lightweight Algorithm to Protect the Web of Things in IoT,” in Proc. Int. Conf. Emerging Technol. Trends Internet Things Comput., pp. 46–60, Springer, Cham, 2021, doi.org/10.1007/978-3-030-97255-4_4.

S. I. Imtiaz, S. ur Rehman, A. R. Javed, Z. Jalil, X. Liu and W. S. Alnumay, “DeepAMD: Detection and Identification of Android Malware Using High-Efficient Deep Artificial Neural Network,” Future Gener. Comput. Syst., vol. 115, pp. 844–856, 2021, doi.org/10.1016/j.future.2020.10.008.

S. Jeon and J. Moon, “Malware-Detection Method with a Convolutional Recurrent Neural Network Using Opcode Sequences,” Information Sci., vol. 535, pp. 1–15, 2020,doi.org/10.1016/j.ins.2020.05.026.

S. Berrios, D. Leiva, B. Olivares, H. Allende-Cid and P. Hermosilla, “Systematic Review: Malware Detection and Classification in Cybersecurity,” Applied Sciences, vol. 15, no. 14, p. 7747, 2025, doi.org/10.3390/app15147747.

U. E. H. Tayyab, F. B. Khan, M. H. Durad, A. Khan and Y. S. Lee, “A Survey of the Recent Trends in Deep Learning Based Malware Detection,” J. Cybersecurity Privacy, vol. 2, no. 4, pp. 800–829, 2022., http://doi.org/10.3390/jcp2040041.‏

W. Qiang, L. Yang and H. Jin, “Efficient and Robust Malware Detection Based on Control Flow Traces Using Deep Neural Networks,” Computers & Security, vol. 122, p. 102871, 2022, doi.org/10.1016/j.cose.2022.102871.

F. A. Aboaoja, A. Zainal, F. A. Ghaleb, B. A. S. Al-Rimy, T. A. E. Eisa and A. A. H. Elnour, “Malware Detection Issues, Challenges, and Future Directions: A Survey,” Applied Sciences, vol. 12, no. 17, p. 8482, 2022, doi.org/10.3390/app12178482.

Y. M. Mohialden et al., “Enhancing security and privacy in healthcare with generative artificial intelligence-based detection and mitigation of data poisoning attacks software,” Jordan Med. J., vol. 58, no. 4, 2024, doi.org/10.35516/jmj.v58i3.2712.

M. S. Mahdi, “Innovative Neural Network Architecture for Progressive Windows Malware Detection via Adaptive Feature Fusion and Multi-stage Learning,” in Proc. Int. Conf. Cybersecurity and Artificial Intelligence Strategies, Springer Nature Switzerland, Cham, pp. 107–121, 2025, doi.org/10.1007/978-3-032-07244-3_7.

S. Wang, Z. Chen, Q. Yan, K. Ji, L. Peng, B. Yang and M. Conti, “Deep and Broad URL Feature Mining for Android Malware Detection,” Information Sci., vol. 513, pp. 600–613, 2020, doi.org/10.1016/j.ins.2019.11.008.

F. A. Abdulazeez, I. T. Ahmed and B. T. Hammad, “Examining the Performance of Various Pretrained Convolutional Neural Network Models in Malware Detection,” Appl. Sci., vol. 14, no. 6, p. 2614, 2024, doi.org/10.3390/app14062614.

H. Babbar, S. Rani and W. Boulila, “NGMD: Next Generation Malware Detection in Federated Server with Deep Neural Network Model for Autonomous Networks,” Sci. Rep., vol. 14, p. 10898, 2024, doi.org/10.1038/s41598-024-61298-7.

E. Kabanda, “Performance of Machine Learning and Other Artificial Intelligence Paradigms in Cybersecurity,” Oriental J. Comput. Sci. Technol., vol. 13, pp. 1–9, 2020,doi.org/10.13005/ojcst13.01.01.

S. Latif, N. Ben Said, Z. Idrees, M. Frikha and H. Chaieb, “A Novel Attack Detection Scheme for the Industrial Internet of Things Using a Lightweight Random Neural Network,” IEEE Access, vol. 8, pp. 91065–91074, 2020, doi.org/10.1109/ACCESS.2020.2994079.

I. H. Sarker, “Deep Cybersecurity: A Comprehensive Overview from Neural Network and Deep Learning Perspective,” SN Comput. Sci., vol. 2, pp. 1–14, 2021, doi.org/10.1007/s42979-021-00535- .

P. Yamcharoen, O. Folorunsho, and A. Bayewu , “Application of Reactive Artificial Intelligence Model to Predict Malicious Activities,” Adv. Multidiscip. Sci. Res. J., vol. 9, pp. 45–60, 2021, doi.org/10.22624/AIMS/MATHS/V9N2P5.

M. Alharbi, A. M. El-Sherbeeny and M. A. El-Meligy, “Analyzing the Impact of Cybersecurity-Related Attributes for Intrusion Detection Systems,” Sustainability, vol. 13, pp. 1–15, 2021, https://doi.org/10.3390/su132212337.

M. Abdullahi, A. Bustamam, S. Muhsin and B. M. Ali, “Detecting Cybersecurity Attacks in Internet of Things Using Artificial Intelligence Methods: A Systematic Literature Review,” Electronics, vol. 11, pp. 1–20, 2022, https://doi.org/10.3390/electronics11020198.

F. Algorain and J. Clark, “Bayesian Hyper-Parameter Optimisation for Malware Detection,” Electronics, vol. 11, no. 10, p. 1640, 2022, doi.org/10.3390/electronics11101640.

K. Komarudin, S. Sunardi and Y. Wihardi, “Exploring the Effectiveness of Artificial Intelligence in Detecting Malware and Improving Cybersecurity in Computer Networks,” Eduvest - J. Univers. Stud., vol. 3, pp. 1–12, 2023, http://doi.org/10.59188/eduvest.v3i4.793.

S. M. A. Rizvi, “Enhancing Cybersecurity: The Power of Artificial Intelligence in Threat Detection and Prevention,” Int. J. Adv. Eng. Res. Sci., vol. 10, pp. 115–123, 2023, http://doi.org/10.22161/ijaers.105.8.

R. Ojha, “Use of Artificial Neural Networks to Detect and Prevent Cybersecurity Threats,” NPRC J. Multidiscip. Res., vol. 1, pp. 32–45, 2024, http://doi.org/10.3126/nprcjmr.v1i6.71754.

S. Shahana, “AI-Driven Cybersecurity: Balancing Advancements and Safeguards,” J. Comput. Sci. Technol. Stud., vol. 6, pp. 129–140, 2024, http://doi.org/10.32996/jcsts.2024.6.2.9.

D. Syeda and M. Asghar, “Dynamic Malware Classification and API Categorisation of Windows Portable Executable Files Using Machine Learning,” Applied Sciences, vol. 14, no. 3, p. 1015, 2024, doi.org/10.3390/app14031015

D. Hoogla, “EMBER: 2018 Dataset for Training Static PE Malware Machine Learning Models,” Kaggle, 2018. [Online]. Available: https://www.kaggle.com/datasets/dhoogla/ember-2018-v2-features.

D. G. Corlatescu, A. Dinu, M. P. Gaman and P. Sumedrea, “EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis,” in Adv. Neural Inf. Process. Syst., vol. 36, pp. 26722–26743, 2023.

R. Harang, E. M. Rudd et al., “SOREL-20M: A Large-Scale Benchmark Dataset for Malicious PE Detection,” GitHub, 2020. [Online]. Available: https://github.com/sophos/SOREL-20M.

M. S. Mahdi, Z. L. Ali, A. R. Rashid, N. K. Ibrahim and A. W. Abdulghafour, “A Hybrid Deep Learning Model for Facial Emotion Recognition: Combining Multi-Scale Features, Dynamic Attention, and Residual Connections,” in Proc. 13th Int. Conf. Appl. Innov. IT (ICAIIT), vol. 13, no. 2, pp. 69–77, Jun. 2025, doi.org/10.25673/120395.