Autonomous security analysis and penetration testing model based on attack graph and deep Q-learning network

Cheng FAN; Guoqing HU; Taojie DING; Zhanhua ZHANG

doi:10.11959/j.issn.2096-109x.2023091

您当前的位置：

首页 >

文章列表页 >

Autonomous security analysis and penetration testing model based on attack graph and deep Q-learning network

Papers | 更新时间：2024-06-03

- Autonomous security analysis and penetration testing model based on attack graph and deep Q-learning network
- Chinese Journal of Network and Information Security Vol. 9, Issue 6, Pages: 166-175(2023)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2023091
  CLC： TP393
- Online First：2023-12，
  
  Published：25 December 2023
- 稿件说明：
移动端阅览
Cheng FAN, Guoqing HU, Taojie DING, et al. Autonomous security analysis and penetration testing model based on attack graph and deep Q-learning network[J]. Chinese Journal of Network and Information Security, 2023, 9(6): 166-175.
DOI：

Cheng FAN, Guoqing HU, Taojie DING, et al. Autonomous security analysis and penetration testing model based on attack graph and deep Q-learning network[J]. Chinese Journal of Network and Information Security, 2023, 9(6): 166-175. DOI： 10.11959/j.issn.2096-109x.2023091.

摘要

随着网络技术的快速发展和广泛应用，网络安全问题日益突出，渗透测试成为评估和提升网络安全性的重要手段。然而，传统的人工渗透测试方法效率较低，且易受到人为错误和测试人员技能水平的影响，造成测试结果不确定性大、评估效果不理想等问题。针对以上人工渗透测试中存在的问题，提出了基于攻击图和深度 Q 学习网络（DQN，deep Q-learning network）的自动化安全分析与渗透测试（ASAPT， autonomous security analysis and penetration testing）模型。该模型由训练数据构建和模型训练两部分构成。在训练数据构建阶段，采用攻击图对目标网络进行威胁建模，将网络中存在的漏洞和攻击者可能的攻击路径转化为节点、边，随后结合CVSS（common vulnerability scoring system）漏洞信息库构建对应的“状态-动作”转移矩阵，用以描述攻击者在不同状态下的攻击行为和转移概率，并全面反映攻击者的攻击能力和网络的安全状况。为进一步降低计算复杂度，创新性地使用深度优先搜索算法对转移矩阵进行简化，查找并保留所有能达到最终目标的攻击路径，以便于后续模型训练。在模型训练阶段，使用基于 DQN 的深度强化学习算法对渗透测试中的最优攻击路径进行确定，该算法通过不断与环境交互、更新 Q 值函数，从而逐步优化攻击路径选择。仿真结果表明，ASAPT 模型在最优路径寻找方面准确率可达 84%，收敛速度快，并且在面对大规模网络环境时，相较于传统Q学习具有更好的适应性，能够为实际的渗透测试提供指导。

Abstract

With the continuous development and widespread application of network technology

network security issues have become increasingly prominent.Penetration testing has emerged as an important method for assessing and enhancing network security.However

traditional manual penetration testing methods suffer from inefficiency

human error

and tester skills

leading to high uncertainty and poor evaluation results.To address these challenges

an autonomous security analysis and penetration testing framework called ASAPT was proposed

based on attack graphs and deep Q-learning networks (DQN).The ASAPT framework was consisted of two main components:training data construction and model training.In the training data construction phase

attack graphs were utilized to model the threats in the target network by representing vulnerabilities and possible attacker attack paths as nodes and edges.By integrating the common vulnerability scoring system (CVSS) vulnerability database

a “state-action”transition matrix was constructed

which depicted the attacker’s behavior and transition probabilities in different states.This matrix comprehensively captured the attacker’s capabilities and network security status.To reduce computational complexity

a depth-first search (DFS) algorithm was innovatively applied to simplify the transition matrix

identifying and preserving all attack paths that lead to the final goal for subsequent model training.In the model training phase

a deep reinforcement learning algorithm based on DQN was employed to determine the optimal attack path during penetration testing.The algorithm interacted continuously with the environment

updating the Q-value function to progressively optimize the selection of attack paths.Simulation results demonstrate that ASAPT achieves an accuracy of 84% in identifying the optimal path and exhibits fast convergence speed.Compared to traditional Q-learning

ASAPT demonstrates superior adaptability in dealing with large-scale network environments

which could provide guidance for practical penetration testing.

关键词

Keywords

references

JAJODIA S , NOELS . Topological vulnerability analysis:a powerful new approach for network attack prevention,detection,and response [M ] // Algorithms,Architectures and Information Systems Security . World Scientific , 2008 : 285 - 305 .

网络安全人才实战能力白皮书 [EB ] . 2022 .

White paper on practical competence of cybersecurity talents [EB ] . 2022 .

WANG G Y , WANG H M , CHEN Z J , et al . Research on computer network attack modeling based on attack graph [J ] . Journal of National University of Defense Technology , 2009 , 31 ( 4 ): 74 - 80 .

SHEYNERO , HAINESJ JHA S . Automated generation and analysis of attack graphs [C ] // Proceedings 2002 IEEE Symposium on Security and Privacy . 2004 : 273 - 284 .

OBESJL , SARRAUTEC RICHARTE G . Attack planning in the real world [EB ] .

SARRAUTE C , RICHARTE G , OBES J L . An algorithm to find optimal attack paths in nondeterministic scenarios [C ] // Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence . 2011 : 71 - 80 .

SARRAUTE C , BUFFET O , HOFFMANN J . Penetration testing==POMDP solving [J ] . Computer Science , 2013 , 33 ( 5 ): 535 - 540 .

JONATHON S , HANNA KI . Autonomous penetration testing using reinforcement learning [J ] . arxiv preprint arxiv:1905.05965 , 2019 .

YOUSEFI M , NHAMO M , ZHANGY , . A reinforcement learning approach for attack graph analysis [C ] // 17th IEEE International Conference on Trust,Security and Privacy in Computing and Communications/12th IEEE International Conference on Big Data Science and Engineering (TrustCom/BigDataSE) . 2018 . 212 - 217 .

赵海妮 , 焦健 . 基于强化学习的渗透路径推荐模型 [J ] . 计算机应用 , 2022 , 42 ( 6 ): 1689 - 1694 .

ZHAO H N , JIAO J . Recommendation model of penetration path based on reinforcement learning [J ] . Journal of Computer Application , 2022 , 42 ( 6 ): 1689 - 1694 .

SECURITY A . Acunetix Web application security blog:statixtics from 10,000 leaked hotmail passwords [EB ] .

VYAMAJALA S , MOHD T K , JAVAID A . A real-world Implementation of SQL injection attack using open source tools for enhanced cybersecurity learning [C ] // Proceedings of 2018 IEEE International Conference on Electro/Information Technology(EIT) . 2018 .

康海燕 , 龙墨澜 . 基于吸收马尔可夫链攻击图的网络攻击分析方法研究 [J ] . 通信学报 , 2023 , 44 ( 2 ): 122 - 135 .

KANG H Y , LONG M L . Research on network attack analysis method based on attack graph of absorbing Markov chain [J ] . Journal on Communications , 2023 , 44 ( 2 ): 122 - 135 .

罗智勇 , 宋伟伟 , 张文博 , 等 . 基于 Markov 攻击图和博弈模型的区块链安全态势感知方法 [J ] . 电子与信息学报 , 2023 , 45 ( 4 ): 1374 - 1382 .

LUO Z Y , SONG W W , ZHANG W B , et al . Blockchain security situational awareness method based on Markov attack graph and game model [J ] . Journal of Electronics ＆ Information Technology , 2023 , 45 ( 4 ): 1374 - 1382 .

王赛娥 , 刘彩霞 , 刘树新 . 一种基于攻击图的 5G 网络安全风险评估方法 [J ] . 计算机应用与软件 , 2023 , 40 ( 4 ): 289 - 296 , 335 .

WANG S E , LIU C X , LIU S X . A method of 5G network security risk assessment based on attack graph [J ] . Computer Applications and Software , 2023 , 40 ( 4 ): 289 - 296 , 335 .

KANG J , ZHANG Y , ZHOU Z . A bayesian attack graph-based vulnerability assessment for cyber-physical power systems [J ] . IEEE Transactions on Smart Grid , 2023 , 14 ( 2 ): 139 - 148 .

SABUR A , CHOWDHARY A , HUANG D , et al . S3:a DFW-based scalable security state analysis framework for large-scale data center networks [C ] // The 22nd International Symposium on Research in Attacks,Intrusions and Defenses(RAID) . 2019 : 497 - 485 .

JEFFREY D J , LI J H , CHEN Z S . Reinforcement learning:an introduction [J ] . Neurocomputing , 2000 , 35 ( 1 ).

CHRISTOPHER J . Q-learning [J ] . Machine Learning , 1992 > , 8 ( 3 ): 279 - 292 .

FAN J Q , WANG Z R , XIE Y C , et al . A theoretical analysis of deep q learning [C ] // Proceedings of the 2nd Conference on Learning for Dynamics and Control/Proceedings of Machine Learning Research (PMLR) . 2020 : 486 - 489 .

WANG Y , WANG Y , LI Y , et al . A review on deep Q-learning for robotic control [J ] . IEEE Transactions on Industrial Electronics , 2022 , 69 ( 11 ): 9468 - 9481 .

潘刚 , 米士超 , 郭荣华 , 等 . 基于攻击树和 CVSS 的网络攻击效果评估方法 [J ] . 电子技术应用 , 2022 , 48 ( 4 ): 76 - 80 .

PAN G , MI S C , GUO R H , et al . Evaluation method of network attack effect based on attack tree and CVSS [J ] . Application of Electronic Technique , 2022 , 48 ( 4 ): 76 - 80 .

Views

324

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Semantic communication aware reinforcement learning for communication fault-tolerant UAV collaborative control

Automated Windows domain penetration method based on reinforcement learning

Function approximation method based on weights gradient descent in reinforcement learning

Research on network risk assessment based on attack graph of expected benefits-rate

Key path analysis method for large-scale industrial control network

Related Author

WANG Ran

FENG Bohao

GU Hongyu

ZHANG Yang

Letian SHA

Lige ZHAN

Fu XIAO

Jiankuo DONG

Related Institution

School of Computer Science and Artificial Intelligence Wuhan University of Technology

College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics

Jiangsu Provincial Key Laboratory of Wireless Sensor Network High Technology Research

College of Computer, Nanjing University of Posts and Telecommunications

School of Computer Science and Technology, Soochow University

⁰