基于强化学习的自动化Windows域渗透方法

占力戈; 沙乐天; 肖甫; 董建阔; 张品昌

doi:10.11959/j.issn.2096-109x.2023057

您当前的位置：

首页 >

文章列表页 >

基于强化学习的自动化Windows域渗透方法

学术论文 | 更新时间：2024-06-03

- 基于强化学习的自动化Windows域渗透方法
- Automated Windows domain penetration method based on reinforcement learning
- 网络与信息安全学报 2023年9卷第4期页码：104-120
- 作者机构：
  
  1. 南京邮电大学计算机学院，江苏南京 210023
  2. 江苏省无线传感网高技术研究重点实验室，江苏南京 210023
- 作者简介：
  
  [ "占力戈（1996- ），男，福建南平人，南京邮电大学硕士生，主要研究方向为网络攻防、渗透测试" ]
  [ "沙乐天（1985- ），男，江苏徐州人，博士，南京邮电大学副教授，主要研究方向为网络安全、物联网攻防" ]
  [ "肖甫（1980- ），男，湖南邵阳人，博士，南京邮电大学教授、博士生导师，主要研究方向为传感网、物联网" ]
  [ "董建阔（1992- ），男，河北邢台人，博士，南京邮电大学讲师，主要研究方向为密码工程、高性能并行计算" ]
  [ "张品昌（1985- ），男，安徽临泉人，博士，南京邮电大学讲师，主要研究方向为网络安全、物理层认证及卫星互联网安全" ]
- 基金信息：
  
  国家重点研发计划;The National Key Research and Development Program of China(2018YFB0803400);国家杰出青年科学基金;The National Science Fund for Distinguished Young Scholars of China(62125203);国家自然科学基金;The National Natural Science Foundation of China(62072253)
- DOI：10.11959/j.issn.2096-109x.2023057
  中图分类号：
- 网络首发：2023-08，
  
  纸质出版：2023-08-25
- 稿件说明：
移动端阅览
占力戈, 沙乐天, 肖甫, 等. 基于强化学习的自动化Windows域渗透方法[J]. 网络与信息安全学报, 2023,9(4):104-120.

Lige ZHAN, Letian SHA, Fu XIAO, et al. Automated Windows domain penetration method based on reinforcement learning[J]. Chinese Journal of Network and Information Security, 2023, 9(4): 104-120.
占力戈, 沙乐天, 肖甫, 等. 基于强化学习的自动化Windows域渗透方法[J]. 网络与信息安全学报, 2023,9(4):104-120. DOI： 10.11959/j.issn.2096-109x.2023057.

Lige ZHAN, Letian SHA, Fu XIAO, et al. Automated Windows domain penetration method based on reinforcement learning[J]. Chinese Journal of Network and Information Security, 2023, 9(4): 104-120. DOI： 10.11959/j.issn.2096-109x.2023057.

摘要

Windows 域为用户之间的资源共享及信息交互提供统一的系统服务，在便利内网管理的同时带来了巨大的安全隐患。近年来，针对域控制器的各式攻击层出不穷，实现自动化渗透能够灵活检测 Windows域中存在的漏洞威胁，保障办公网络安全稳定地持久运行，其核心是高效挖掘环境内可行的攻击路径。为此，将渗透测试过程进行强化学习建模，通过智能体与域环境的真实交互发现漏洞组合，进而验证有效的攻击序列；基于主机对渗透进程的贡献差异，减少强化学习模型中非必要的状态与动作，优化路径选择策略，提升实际攻击效率；使用状态动作删减、探索策略优化的 Q 学习算法筛选最优攻击路径，自动验证域环境中所有可能的安全隐患，为域管理员提供防护依据。实验针对典型内网业务场景展开测试，从生成的13 种高效攻击路径中筛选最优路径，通过与相关研究成果对比，突出了所提方法在域控权限获取、主机权限获取、攻击步长、收敛性以及时间代价等方面的性能优化效果。

Abstract

Windows domain provides a unified system service for resource sharing and information interaction among users.However

this also introduces significant security risks while facilitating intranet management.In recent years

intranet attacks targeting domain controllers have become increasingly prevalent

necessitating automated penetration testing to detect vulnerabilities and ensure the ongoing maintenance of office network operations.Then efficient identification of attack paths within the domain environment is crucial.The penetration process was first modeled using reinforcement learning

and attack paths were then discovered and verified through the interaction of the model with the domain environment.Furthermore

unnecessary states in the reinforcement learning model were trimmed based on the contribution differences of hosts to the penetration process

aiming to optimize the path selection strategy and improve the actual attack efficiency.The Q-learning algorithms with solution space refinement and exploration policy optimization were utilized to filter the optimal attack path.By employing this method

all security threats in the domain can be automatically verified

providing a valuable protection basis for domain administrators.Experiments were conducted on typical Windows domain scenarios

and the results show that the optimal path is selected from the thirteen efficient paths generated by the proposed method

while also providing better performance optimization in terms of domain controller intrusion

domain host intrusion

attack steps

convergence

and time cost compared to other approaches.

关键词

Keywords

references

尹圣超 . Windows域攻防关键技术研究 [D ] . 西安:西安电子科技大学 , 2021 .

YIN Z C . Research on key technologies of attack and defense of Windows domain [D ] . Xi’an:Xidian University , 2021 .

SHEBLI H , BEHESHTI B D . A study on penetration testing process and tools [C ] // 2018 IEEE Long Island Systems,Applications and Technology Conference (LISAT) , 2018 : 1 - 7 .

STEFINKO Y , PISKOZUB A , BANAKH R . Manual and automated penetration testing,benefits and drawbacks,modern tendency [C ] // 2016 13th International Conference on Modern Problems of Radio Engineering,Telecommunications and Computer Science(TCSET), . 2016 : 488 - 491 .

ALMUBAIRIK N A , WILLS G . Automated penetration testing based on a threat model [C ] // 2016 11th International Conference for Internet Technology and Secured Transactions (ICITST) . 2016 : 413 - 414 .

王硕 , 汤光明 , 寇广 , 等 . 基于因果知识网络的攻击路径预测方法 [J ] . 通信学报 , 2016 , 37 ( 10 ): 188 - 198 .

WANG S , TANG G M , KOU G , et al . Attack path prediction method based on causal knowledge net [J ] . Journal on Communications , 2016 , 37 ( 10 ): 188 - 198 .

王硕 , 王建华 , 汤光明 , 等 . 一种智能高效的最优渗透路径生成方法 [J ] . 计算机研究与发展 , 2019 , 56 ( 5 ): 929 - 941 .

WANG S , WANG J H , TANG G M , et al . Intelligent and efficient method for optimal penetration path generation [J ] . Journal of Computer Research and Development , 2019 , 56 ( 5 ): 929 - 941 .

高阳 , 陈世福 , 陆鑫 . 强化学习研究综述 [J ] . 自动化学报 , 2004 , 30 ( 1 ): 86 - 100 .

GAO Y , CHEN S F , LU X . Research on reinforcement learning technology:a review [J ] . ACTA Automatica Sinica , 2004 , 30 ( 1 ): 86 - 100 .

WATKINS C J C H , DAYAN P . Q-learning [J ] . Machine Learning , 1992 , 8 ( 3-4 ): 279 - 292 .

SCHNEIER B . Attack trees-modeling security threats [J ] . Drdobbs Journal , 1999 , 24 ( 12 ): 21 - 29 .

SHEYNER O , HAINES J , JHA S , et al . Automated generation and analysis of attack graphs [C ] // Proceedings 2002 IEEE Symposium on Security and Privacy . 2002 : 273 - 284 .

叶子维 , 郭渊博 , 王宸东 , 等 . 攻击图技术应用研究综述 [J ] . 通信学报 , 2017 , 38 ( 11 ): 121 - 132 .

YE Z W , GUO Y B , WANG C D , et al . Survey on application of attack graph technology [J ] . Journal on Communications , 2017 , 38 ( 11 ): 121 - 132 .

陈锋 , 张怡 , 苏金树 , 等 . 攻击图的两种形式化分析 [J ] . 软件学报 , 2010 , 21 ( 4 ): 838 - 848 .

CHEN F , ZHANG Y , SU J S , et al . Two formal analyses of attack graphs [J ] . Journal of Software , 2010 , 21 ( 4 ): 838 - 848 .

YOUSEFI M , MTETWA N , Zhang Y , et al . A reinforcement learning approach for attack graph analysis [C ] // 2018 17th IEEE International Conference On Trust,Security And Privacy In Computing And Communications/12th IEEE International Conference on Big Data Science And Engineering (TrustCom/BigDataSE) . 2018 : 212 - 217 .

张蕾 , 崔勇 , 刘静 , 等 . 机器学习在网络空间安全研究中的应用 [J ] . 计算机学报 , 2018 , 41 ( 9 ): 1943 - 1975 .

ZHANG L , CUI Y , LIU J , et al . Application of machine learning in cyberspace security research [J ] . Chinese Journal of Computers , 2018 , 41 ( 9 ): 1943 - 1975 .

OU X , GOVINDAVAJHALA S , APPEL A W . MulVAL:a logic-based network security analyzer [C ] // USENIX Security Symposium . 2005 : 113 - 128 .

HU Z , BEURAN R , TAN Y . Automated penetration testing using deep reinforcement learning [C ] // 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS ＆ PW) . 2020 : 2 - 10 .

刘全 , 翟建伟 , 章宗长 , 等 . 深度强化学习综述 [J ] . 计算机学报 , 2018 , 41 ( 1 ): 1 - 27 .

LIU Q , ZHAI J W , ZHANG Z C , et al . A survey on deep reinforcement learning [J ] . Chinese Journal of Computers , 2018 , 41 ( 1 ): 1 - 27 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

周飞燕 , 金林鹏 , 董军 . 卷积神经网络研究综述 [J ] . 计算机学报 , 2017 , 40 ( 6 ): 1229 - 1251 .

ZHOU F Y , JIN L P , DONG J . Review of convolutional neural network [J ] . Chinese Journal of Computers , 2017 , 40 ( 6 ): 1229 - 1251 .

LI Y , LI X . Research on multi-Target network security assessment with attack graph expert system model [J ] . Scientific Programming , 2021 ,( 3 ): 1 - 11 .

OBES J L , SARRAUTE C , RICHARTE G . Attack planning in the real world [C ] // Working Notes for the 2010 AAAI Workshop on Intelligent Security (SecArt) . 2010 :10.

HASLUM P , LIPOVETZKY N , MAGAZZENI D , et al . An introduction to the planning domain definition language [J ] . Synthesis Lectures on Artificial Intelligence and Machine Learning , 2019 , 13 ( 2 ): 1 - 187 .

臧艺超 , 周天阳 , 朱俊虎 , 等 . 领域独立智能规划技术及其面向自动化渗透测试的攻击路径发现研究进展 [J ] . 电子与信息学报 , 2020 , 42 ( 9 ): 2095 - 2107 .

ZANG Y C , ZHOU T Y , ZHU J H , et al . Domain-Independent intelligent planning technology and its application to automated penetration testing oriented attack path discovery [J ] . Journal of Electronics and Information Technology , 2020 , 42 ( 9 ): 2095 - 2107 .

SARRAUTE C , RICHARTE G , LUCÁNGELI O J . An algorithm to find optimal attack paths in nondeterministic scenarios [C ] // Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence . 2011 : 71 - 80 .

SOHN S , OH J , LEE H . Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies [C ] // The 32nd Conference on Neural Information Processing Systems . 2018 : 7156 - 7166 .

DE SILVA L , PADGHAM L , SARDINA S . HTN-like solutions for classical planning problems:an application to BDI agent systems [J ] . Theoretical Computer Science , 2019 , 763 : 12 - 37 .

MOHR F , WEVER M , HÜLLERMEIER E . ML-plan:automated machine learning via hierarchical planning [J ] . Machine Learning , 2018 , 107 ( 8-10 ): 1495 - 1515 .

Core impact [J ] . SC Magazine:The International Journal of Computer Security , 2010 ( 2 ): 21 .

HOFFMANN J . The metric-FF planning system:translating “ignoring delete lists” to numeric state variables [J ] . Journal of Artificial Intelligence Research , 2011 , 20 : 291 - 341 .

范长杰 . 基于马尔可夫决策理论的规划问题的研究 [D ] . 合肥:中国科学技术大学 , 2008 .

FAN C J . Research on planning based on Markov decision theory [D ] . Hefei:University of Science and Technology of China , 2008 .

赵海妮 , 焦健 . 基于强化学习的渗透路径推荐模型 [J ] . 计算机应用 , 2022 : 1 - 7 .

ZHAO H N , JIAO J . Infiltration path recommendation model based on reinforcement learning [J ] . Journal of Computer Applications , 2022 : 1 - 7 .

DURKOTA K , LISÝ V . Computing optimal policies for attack graphs with action failures and costs [C ] // The 7th European Starting AI Researcher Symposium . 2014 : 101 - 110 .

SHMARYAHU D , SHANI G , HOFFMANN J , et al . Constructing plan trees for simulated penetration testing [C ] // The 26th International Conference on Automated Planning and Scheduling . 2016 :121.

KAELBLING L P , LITTMAN M L , CASSANDRA A R . Planning and acting in partially observable stochastic domains [J ] . Artificial Intelligence , 1998 , 101 ( 1-2 ): 99 - 134 .

LIU B B , KANG Y , JIANG X F , et al . A fast approximation method for partially observable Markov decision processes [J ] . Journal of Systems Science and Complexity , 2018 , 31 ( 6 ): 1423 - 1436 .

王作广 , 魏强 , 刘雯雯 . 基于攻击树与 CVSS 的工业控制系统风险量化评估 [J ] . 计算机应用研究 , 2016 , 33 ( 12 ): 3785 - 3790 .

WANG ZG , WEI Q , LIU W W . Quantitative risk assessment of industrial control systems based on attack trees and CVSS [J ] . Application Research of Computers , 2016 , 33 ( 12 ): 3785 - 3790 .

刘奇旭 , 张翀斌 , 张玉清 , 等 . 安全漏洞等级划分关键技术研究 [J ] . 通信学报 , 2012 , 33 ( S1 ): 79 - 87 .

LIU Q X , ZHANG C B , ZHANG Y Q , et al . Research on the key technology of security vulnerability threat classification [J ] . Journal on Communications , 2012 , 33 ( S1 ): 79 - 87 .

AonCyberLabs . Windows-exploit-suggester [EB ] .

byt3bl33d3r . DeathStar [EB ] .

浏览量

386

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

语义通信下基于强化学习的无人机通信容错协同控制

基于攻击图和深度Q学习网络的自动化安全分析与渗透测试模型

强化学习中基于权重梯度下降的函数逼近方法

基于期望收益率攻击图的网络风险评估研究

基于强化学习的低时延车联网群密钥分配管理技术