基于模仿学习的深度强化学习训练数据推断攻击

陈晋音; 瞿康赟; 郑海斌

doi:10.11959/j.issn.2096-109x.2025060

您当前的位置：

首页 >

文章列表页 >

基于模仿学习的深度强化学习训练数据推断攻击

专题：大模型数据安全与隐私保护 | 更新时间：2025-11-08

- 基于模仿学习的深度强化学习训练数据推断攻击
- Membership inference attack based on imitation learning for deep reinforcement learning
- 网络与信息安全学报 2025年11卷第5期页码：50-67
- 作者机构：
  
  1.浙江工业大学网络空间安全研究院，浙江杭州 310023
  2.浙江工业大学信息工程学院，浙江杭州 310023
- 作者简介：
  
  [ "陈晋音（1982- ），女，宁波象山人，浙江工业大学教授，主要研究方向为人工智能安全、图数据挖掘和进化计算。" ]
  [ "瞿康赟（2001- ），男，安徽天长人，浙江工业大学硕士生，主要研究方向为人工智能安全、深度学习和强化学习。" ]
  [ "郑海斌（1995- ），男，浙江台州人，浙江工业大学讲师，主要研究方向为深度学习、人工智能安全和图像识别。" ]
- 基金信息：
  
  国家自然科学基金(62072406;62406286);浙江省自然科学基金(LDQ23F020001);浙江省重点研发计划(2022C01018);国家重点研发计划(2018AAA0100801)
- DOI：10.11959/j.issn.2096-109x.2025060
  中图分类号： TP393
- 收稿：2025-05-13，
  
  修回：2025-08-16，
  
  纸质出版：2025-10-25
- 稿件说明：
移动端阅览
陈晋音, 瞿康赟, 郑海斌. 基于模仿学习的深度强化学习训练数据推断攻击[J]. 网络与信息安全学报, 2025, 11(5): 50-67.

CHEN J Y, QU K Y, ZHENG H B. Membership inference attack based on imitation learning for deep reinforcement learning[J]. Chinese Journal of Network and Information Security, 2025, 11(5): 50-67.
陈晋音, 瞿康赟, 郑海斌. 基于模仿学习的深度强化学习训练数据推断攻击[J]. 网络与信息安全学报, 2025, 11(5): 50-67. DOI： 10.11959/j.issn.2096-109x.2025060.

CHEN J Y, QU K Y, ZHENG H B. Membership inference attack based on imitation learning for deep reinforcement learning[J]. Chinese Journal of Network and Information Security, 2025, 11(5): 50-67. DOI： 10.11959/j.issn.2096-109x.2025060.

摘要

深度强化学习凭借其卓越的性能优势，在PC游戏、机器人控制、自动驾驶等领域获得了广泛应用。该技术通常依赖大量的训练数据，通过不断试错提升智能体的决策能力。已有研究指出，深度强化学习存在训练数据泄露的隐私风险，但相关攻击方法通常对过拟合的智能体更加有效。为此，针对深度强化学习训练得到且具有泛化能力的智能体，提出一种基于模仿学习的目标场景隐私攻击方法（trajectory informed privacy attack strategy

TIPAS）。该方法通过分析智能体的行为轨迹，采用基于行为克隆的模仿学习方法来构建影子智能体模型，并利用影子智能体在多个场景中生成行为策略特征，结合集成学习训练攻击模型，实现在黑盒访问训练策略场景下精确推断出目标智能体的隐私信息。最后，分别在4种网络拓扑场景和4种算法模型上进行了实验，并以基于奖励均值与方差特征的支持向量机方法，以及未使用集成学习的单一多层感知机模型作为对比基准，通过推断准确率的指标对比验证了所提方法在攻击性能上的优越性。实验证明，所提方法在Gym平台的4种经典游戏场景中利用差分隐私和正则化作为隐私保护方法的防御条件，均具有优秀的攻击性能。

Abstract

Deep reinforcement learning (DRL)

owing to its remarkable performance advantages

had been widely applied in fields such as game intelligence

robotic control

and autonomous driving. This technology typically relied on large amounts of training data

and the agent’s decision-making ability was improved through continuous trial and error. Previous studies had indicated that DRL faced privacy risks due to potential training data leakage; however

existing attack methods were generally found to be more effective against overfitted agents. To address this limitation

a trajectory-informed privacy attack strategy (TIPAS) based on imitation learning was proposed for DRL-trained agents with strong generalization ability. In this method

the agent's behavioral trajectories were analyzed

and a shadow agent model was constructed using behavior cloning. The behavioral policy features generated by the shadow agent in multiple environments were then integrated through ensemble learning to train an attack model

which enabled accurate inference of private information from the target agent under black-box access to the training policy. Experiments were conducted across four network topology scenarios and four algorithmic models. The proposed method was compared against two baselines: a support vector machine utilizing reward mean and variance features

and a single multilayer perceptron model without ensemble learning. The comparison of inference accuracy demonstrated the superiority of the proposed approach in attack performance. Experimental results show that excellent attack performance is achieved by the proposed method across four classic game environments on the Gym platform

even under defensive conditions where differential privacy and regularization techniques are employed.

关键词

Keywords

references

TAN R J , ZHOU J , DU H B , et al . An modeling processing method for video games based on deep reinforcement learning [C ] // Proceedings of the 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC) . Piscataway : IEEE Press , 2019 : 939 - 942 .

LIU R R , NAGEOTTE F , ZANNE P , et al . Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review [J ] . Robotics , 2021 , 10 ( 1 ): 22 .

IBARZ J , TAN J , FINN C , et al . How to train your robot with deep reinforcement learning: lessons we have learned [J ] . The International Journal of Robotics Research , 2021 , 40 ( 4/5 ): 698 - 721 .

PALANISAMY P . Multi-agent connected autonomous driving using deep reinforcement learning [C ] // Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN) . Piscataway : IEEE Press , 2020 : 1 - 7 .

ASHRAF N M , MOSTAFA R R , SAKR R H , et al . Optimizing hyperparameters of deep reinforcement learning for autonomous driving based on whale optimization algorithm [J ] . PLoS One , 2021 , 16 ( 6 ): e0252754 .

FENG S , SUN H W , YAN X T , et al . Dense reinforcement learning for safety validation of autonomous vehicles [J ] . Nature , 2023 , 615 ( 7953 ): 620 - 627 .

刘圣龙 , 黄秀丽 , 江伊雯 , 等 . 面向多方数据融合分析的隐私计算技术综述 [J ] . 网络与信息安全学报 , 2024 , 10 ( 6 ): 24 - 36 .

LIU S L , HUANG X L , JIANG Y W , et al . Review of privacy computing techniques for multi-party data fusion analysis [J ] . Chinese Journal of Network and Information Security , 2024 , 10 ( 6 ): 24 - 36 .

HENDERSON P , ISLAM R , BACHMAN P , et al . Deep reinforcement learning that matters [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Hong Kong : AAAI Press , 2018 : 3207 - 3214 .

COBBE K , KLIMOV O , HESSE C , et al . Quantifying generalization in reinforcement learning [C ] // Proceedings of the International Conference on Machine Learning . New York : PMLR , 2019 : 1282 - 1289 .

PAN X , WANG W , ZHANG X , et al . How you act tells a lot: privacy-leaking attack on deep reinforcement learning [C ] // Proceedings of the 18th International Conference on Autonomous Agents and Multi-Agent Systems . Berlin : Springer , 2019 : 368 - 376 .

BOJARSKI M , DEL TESTA D , DWORAKOWSKI D , et al . End to end learning for self-driving cars [J ] . arXiv Preprint , arXiv: 1604.07316 , 2016 .

KENDALL A , HAWKE J , JANZ D , et al . Learning to drive in a day [J ] . arXiv Preprint , arXiv: 1807.00412 , 2018 .

KIRAN B R , SOBH I , TALPAERT V , et al . Deep reinforcement learning for autonomous driving: a survey [J ] . IEEE Transactions on Intelligent Transportation Systems , 2022 , 23 ( 6 ): 4909 - 4926 .

NG A Y , RUSSELL S . Algorithms for inverse reinforcement learning [C ] // Proceedings of the Seventeenth International Conference on Machine Learning . Morgan Kaufmann : San Francisco , 2000 : 663 - 670 .

FINN C , LEVINE S , ABBEEL P . Guided cost learning: deep inverse optimal control via policy optimization [C ] // Proceedings of the 33nd International Conference on Machine Learning . Cambridge : MIT Press , 2016 : 49 - 58 .

CODEVILLA F , MÜLLER M , LÓPEZ A , et al . End-to-end driving via conditional imitation learning [C ] // Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA) . Piscataway : IEEE Press , 2018 : 4693 - 4700 .

王晓芳 , 顾焜仁 . 一种深度强化学习与模仿学习结合的突防策略 [J ] . 宇航学报 , 2023 , 44 ( 6 ): 914 - 925 .

WANG X F , GU K R . A penetration strategy combining deep reinforcement learning and imitation learning [J ] . Journal of Astronautics , 2023 , 44 ( 6 ): 914 - 925 .

FREEMAN C D , METZ L , HA D . Learning to predict without looking ahead: world models without forward prediction [C ] // Proceedings of the 32nd Conference on Neural Information Processing Systems ( 2019) . Vancouver : NeurIPS , 2019 : 5380 - 5391 .

任奎 , 孟泉润 , 闫守琨 , 等 . 人工智能模型数据泄露的攻击与防御研究综述 [J ] . 网络与信息安全学报 , 2021 , 7 ( 1 ): 1 - 10 .

REN K , MENG Q R , YAN S K , et al . Survey of artificial intelligence data security and privacy protection [J ] . Chinese Journal of Network and Information Security , 2021 , 7 ( 1 ): 1 - 10 .

GOMROKCHI M , AMIN S , ABOUTALEBI H , et al . Membership inference attacks against temporally correlated data in deep reinforcement learning [J ] . IEEE Access , 2023 , 11 : 42796 - 42808 .

FRANÇOIS-LAVET V , HENDERSON P , ISLAM R , et al . An introduction to deep reinforcement learning [J ] . Foundations and Trends in Machine Learning , 2018 , 11 ( 3/4 ): 219 - 354 .

POPESCU M , BALAS V E , PERESCU-POPESCU L , et al . Multilayer perceptron and neural networks [J ] . WSEAS Transactions on Circuits and Systems Archive , 2015 , 8 ( 7 ): 579 - 588 .

GIRSANG N D . Classification of batik images using multilayer perceptron with histogram of oriented gradient feature extraction [C ] // Proceeding of International Conference on Science and Engineering . 2021 : 197 - 204 .

王文尚 , 高凡 , 侯冰莹 , 等 . 基于多层感知机的ATM英文凭单识别 [J ] . 信息技术 , 2024 , 48 ( 5 ): 144 - 148, 154 .

WANG W S , GAO F , HOU B Y , et al . English ATM voucher identification based on multilayer perceptron [J ] . Information Technology , 2024 , 48 ( 5 ): 144 - 148, 154 .

ALI-ALNUAIM A , ZAKARIAH M , SHUKLA P K , et al . Human-computer interaction for recognizing speech emotions using multilayer perceptron classifier [J ] . Journal of Healthcare Engineering , 2022 .

NOOR F , IQBAL M J , YAQOOB S , et al . Depression detection in social media using bagging classifier [J ] . Depression , 2023 .

BROCKMAN G , CHEUNG V , PETTERSSON L , et al . Openai gym [J ] . arXiv preprint , arXiv: 1606.01540 , 2016 .

ALMASAN P , SUÁREZ-VARELA J , RUSEK K , et al . Deep reinforcement learning meets graph neural networks: Exploring a routing optimization use case [J ] . Computer Communications , 2022 , 196 : 184 - 194 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Playing atari with deep reinforcement learning [EB ] . arXiv Preprint , arXiv: 1312. 5602 , 2013 .

SALH A , AUDAH L , ALHARTOMI M A , et al . Smart packet transmission scheduling in cognitive IoT systems: DDQN based approach [J ] . IEEE Access , 2022 , 10 : 50023 - 50036 .

BAN T W . An autonomous transmission scheme using dueling DQN for D2D communication networks [J ] . IEEE Transactions on Vehicular Technology , 2020 , 69 ( 12 ): 16348 - 16352 .

SCHULMAN J , WOLSKI F , DHARIWAL P , et al . Proximal policy optimization algorithms [J ] . arxiv preprint , arxiv: 1707.06347 , 2017 .

HAGBERG A , SWART P S , CHULT D . Exploring network structure, dynamics, and function using NetworkX [R ] . Los Alamos National Lab, Los Alamos , 2008 .

KNIGHT S , NGUYEN H X , FALKNER N , et al . The Internet topology zoo [J ] . IEEE Journal on Selected Areas in Communications , 2011 , 29 ( 9 ): 1765 - 1775 .

MIRONOV I . Renyi differential privacy [J ] . arXiv preprint , arXiv:1702. 07476 . 2017 .

SRIVASTAVA N , HINTON G , KRIZHEVSKY A , et al . Dropout: a simple way to prevent neural networks from overfitting [J ] . Journal of Machine Learning Research , 2014 , 15 ( 1 ): 1929 - 1958 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于分布散度的自适应模糊测试优化方法

基于深度强化学习的恶意ELF文件检测对抗方法

基于深度强化学习的高性能导向性模糊测试方案