Adversarial method for malicious ELF file detection based on deep reinforcement learning

SUN He; ZHANG Bocheng; GENG Jiaxuan; WU Di; WANG Junfeng; FANG Zhiyang

doi:10.11959/j.issn.2096-109x.2024074

您当前的位置：

首页 >

文章列表页 >

Adversarial method for malicious ELF file detection based on deep reinforcement learning

Papers | 更新时间：2024-12-20

- Adversarial method for malicious ELF file detection based on deep reinforcement learning
- Chinese Journal of Network and Information Security Vol. 10, Issue 5, Pages: 152-162(2024)
- 作者机构：
  
  1.试验物理与计算数学重点实验室，北京 100094
  2.四川大学网络空间安全学院，四川成都 610207
  3.四川大学计算机学院，四川成都 610065
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(U2133208);The Key Research and Development Program of Science and Technology Department of Sichuan Province(2023YFG0290)
- DOI：10.11959/j.issn.2096-109x.2024074
  CLC： TP309.5
- Received：30 April 2024，
  
  Revised：2024-07-15，
  
  Published：25 October 2024
- 稿件说明：
移动端阅览
孙贺, 张博成, 耿嘉炫, 等. 基于深度强化学习的恶意ELF文件检测对抗方法[J]. 网络与信息安全学报, 2024, 10(5): 152-162.

Format: SUN H, ZHANG B C, GENG J X, et al. Adversarial method for malicious ELF file detection based on deep reinforcement learning[J]. Chinese Journal of Network and Information Security, 2024, 10(5): 152-162.
孙贺, 张博成, 耿嘉炫, 等. 基于深度强化学习的恶意ELF文件检测对抗方法[J]. 网络与信息安全学报, 2024, 10(5): 152-162. DOI： 10.11959/j.issn.2096-109x.2024074.

Format: SUN H, ZHANG B C, GENG J X, et al. Adversarial method for malicious ELF file detection based on deep reinforcement learning[J]. Chinese Journal of Network and Information Security, 2024, 10(5): 152-162. DOI： 10.11959/j.issn.2096-109x.2024074.

摘要

近年来，基于深度学习的恶意可执行与可链接格式（executable and linkable format，ELF）文件检测研究取得了显著进展。同时，关于模型对抗性攻击的研究也得到了广泛关注，攻击者可以通过生成对抗样本误导神经网络，使恶意软件被错误归类为良性软件，从而逃逸检测。尽管已有多种生成对抗样本的方法被提出，但它们通常不适合对ELF文件进行修改，或缺乏在不同检测模型之间迁移的能力。为克服现有方法的局限性，提出了一种基于深度强化学习的恶意ELF文件检测对抗方法，该方法通过构造目标检测模型的最优扰动字节序列，能够在保留ELF文件原有功能的前提下生成对抗样本，而无须依赖目标模型的内部实现细节。实验结果表明，该方法生成的对抗样本针对目标检测模型的逃逸成功率达到76.80%，并能通过对抗训练提升目标检测模型的鲁棒性。

Abstract

In recent years

research on detecting malicious executable and linkable format (ELF) files based on deep learning had made significant progress. At the same time

adversarial attacks on models had also gained widespread attention. Attackers could generate adversarial examples to mislead neural networks

causing malicious software to be misclassified as benign

thereby evading detection. Although various methods for generating adversarial examples had been proposed

they were often not suitable for modifying ELF files or lacked the ability to transfer across different detection models. To overcome the limitations of existing methods

an adversarial example generation method based on deep reinforcement learning was proposed. This method generated adversarial examples by constructing optimal perturbation bytes for the target detection model while preserving the original functionality of the ELF files

without relying on the internal details of the target model. The experimental results showed that the adversarial examples generated by this method achieved a 76.80% success rate in evading the target detection model

and could enhance the robustness of the model through adversarial training.

关键词

Keywords

references

GALANTE L , BOTACIN M , GRÉGIO A , et al . Malicious Linux binaries: a landscape [C ] // Proceedings of the Anais Estendidos do XVIII Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais (SBSeg Estendido 2018). 2018 : 213 - 222 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2016 : 770 - 778 .

AMODEI D , ANUBHAI R , BATTENBERG E , et al . Deep speech 2: end-to-end speech recognition in English and mandarin [EB ] . arXiv preprint arxiv: 1512.02595 , 2015 .

DEVLIN J , CHANG M W , LEE K , et al . BERT: pre-training of deep bidirectional transformers for language understanding [EB ] . arXiv preprint arXiv: 1810.04805 , 2018 .

JEON J , JEONG B , BAEK S , et al . Hybrid malware detection based on Bi-LSTM and SPP-net for smart IoT [J ] . IEEE Transactions on Industrial Informatics , 2022 , 18 ( 7 ): 4830 - 4837 .

WANG H R , HE H , ZHANG W Z . Demadroid: object reference graph-based malware detection in android [J ] . Security and Communication Networks , 2018 : 7064131 .

CORUM A , JENKINS D , ZHENG J . Robust PDF malware detection with image visualization and processing techniques [C ] // Proceedings of the 2019 2nd International Conference on Data Intelligence and Security (ICDIS) . Piscataway : IEEE Press , 2019 : 108 - 114 .

COULL S E , GARDNER C . Activation analysis of a byte-based deep neural network for malware classification [C ] // Proceedings of the 2019 IEEE Security and Privacy Workshops (SPW) . Piscataway : IEEE Press , 2019 : 21 - 27 .

KOLOSNJAJI B , DEMONTIS A , BIGGIO B , et al . Adversarial malware binaries: evading deep learning for malware detection in executables [C ] // Proceedings of the 2018 26th European Signal Processing Conference (EUSIPCO) . Piscataway : IEEE Press , 2018 : 533 - 537 .

DEMETRIO L , BIGGIO B , LAGORIO G , et al . Explaining vulnerabilities of deep learning to adversarial malware binaries [EB ] . arXiv preprint arXiv: 1901.03583 , 2019 .

KREUK F , BARAK A , AVIV-REUVEN S , et al . Deceiving end-to-end deep learning malware detectors using adversarial examples [EB ] . arXiv preprint arXiv: 1802.04528 , 2018 .

DEMETRIO L , COULL S E , BIGGIO B , et al . Adversarial examples [J ] . ACM Transactions on Privacy and Security , 2021 , 24 ( 4 ): 1 - 31 .

RAFF E , BARKER J , SYLVESTER J , et al . Malware detection by eating a whole exe [C ] // Proceedings of the Workshops at the Thirty-Second AAAI Conference on Artificial Intelligence . 2018 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Playing atari with deep reinforcement learning [EB ] . arXiv preprint arXiv: 1312.5602 , 2013 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

SZEGEDY C , ZAREMBA W , SUTSKEVER I , et al . Intriguing properties of neural networks [EB ] . arXiv preprint arXiv: 1312.6199 , 2013 .

GOODFELLOW I J , SHLENS J , SZEGEDY C , et al . Explaining and harnessing adversarial examples [J ] . arXiv preprint arXiv: 1412.6572 , 2014 .

AL-DUJAILI A , HUANG A , HEMBERG E , et al . Adversarial deep learning for robust detection of binary encoded malware [C ] // Proceedings of the 2018 IEEE Security and Privacy Workshops (SPW) . Piscataway : IEEE Press , 2018 : 76 - 82 .

DING Y X , SHAO M M , NIE C , et al . An efficient method for generating adversarial malware samples [J ] . Electronics , 2022 , 11 ( 1 ): 154 .

KOSIKOWSKI A , CHO D , NINAN M , et al . EvilELF: evasion attacks on deep-learning malware detection over ELF files [C ] // Proceedings of the 2023 International Conference on Machine Learning and Applications (ICMLA) . Piscataway : IEEE Press , 2023 : 1702 - 1709 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Membership inference attack based on imitation learning for deep reinforcement learning

Self-adaptive fuzzing optimization method based on distribution divergence

Multi-feature fusion malware detection method based on attention and gating mechanisms

High-performance directional fuzzing scheme based on deep reinforcement learning

Android malware detection method based on deep neural network

Related Author

ZHENG Haibin

QU Kangyun

CHEN Jinyin

ZHANG Chao

MA Zheyu

JI Jiangan

XU Hang

Jianbiao ZHANG

Related Institution

Institute of Cyberspace Security, Zhejiang University of Technology

College of Information Engineering, Zhejiang University of Technology

Key Laboratory of Cyberspace Security, Ministry of Education

Institute for Network Sciences and Cyberspace, Tsinghua University

School of Cyber Science and Engineering, Shanghai Jiao Tong University

⁰