基于鸽群的鲁棒强化学习算法

张明英; 华冰; 张宇光; 李海东; 郑墨泓

doi:10.11959/j.issn.2096-109x.2022064

您当前的位置：

首页 >

文章列表页 >

基于鸽群的鲁棒强化学习算法

专题：大数据与人工智能安全 | 更新时间：2024-06-03

- 基于鸽群的鲁棒强化学习算法
- Robust reinforcement learning algorithm based on pigeon-inspired optimization
- 网络与信息安全学报 2022年8卷第5期页码：66-74
- 作者机构：
  
  1. 中国电子技术标准化研究院，北京 100007
  2. 南京航空航天大学航天学院，江苏南京 211106
  3. 中国电子科技集团公司第七研究所，广东广州 510000
- 作者简介：
  
  [ "张明英（1985- ），男，广西北海人，中国电子技术标准化研究院高级工程师，主要研究方向为人工智能、知识图谱、大数据" ]
  [ "华冰（1978- ），女，江苏南京人，南京航空航天大学副研究员，主要研究方向为飞行器导航、智能数据处理" ]
  [ "张宇光（1991- ），男，内蒙古包头人，中国电子技术标准化研究院工程师，主要研究方向为数据安全、人工智能安全、个人信息保护、计算机视觉、视觉生成" ]
  [ "李海东（1992- ），男，湖北孝感人，中国电子技术标准化研究院工程师，主要研究方向为人工智能安全、大数据安全、个人信息保护" ]
  [ "郑墨泓（1995- ），女，广东潮阳人，中国电子科技集团公司第七研究所助理工程师，主要研究方向为航天器姿态控制、无人机组网" ]
- 基金信息：
  
  科技创新2030重大项目;Science and Technology Innovation 2030 Major Project(2020AAA0107804)
- DOI：10.11959/j.issn.2096-109x.2022064
  中图分类号： TP393
- 网络首发：2022-10，
  
  纸质出版：2022-10-15
- 稿件说明：
移动端阅览
张明英, 华冰, 张宇光, 等. 基于鸽群的鲁棒强化学习算法[J]. 网络与信息安全学报, 2022,8(5):66-74.

Mingying ZHANG, Bing HUA, Yuguang ZHANG, et al. Robust reinforcement learning algorithm based on pigeon-inspired optimization[J]. Chinese Journal of Network and Information Security, 2022, 8(5): 66-74.
张明英, 华冰, 张宇光, 等. 基于鸽群的鲁棒强化学习算法[J]. 网络与信息安全学报, 2022,8(5):66-74. DOI： 10.11959/j.issn.2096-109x.2022064.

Mingying ZHANG, Bing HUA, Yuguang ZHANG, et al. Robust reinforcement learning algorithm based on pigeon-inspired optimization[J]. Chinese Journal of Network and Information Security, 2022, 8(5): 66-74. DOI： 10.11959/j.issn.2096-109x.2022064.

摘要

强化学习是一种人工智能算法，具有计算逻辑清晰、模型易扩展的优点，可以在较少甚至没有先验信息的前提下，通过和环境交互并最大化值函数，调优策略性能，有效地降低物理模型引起的复杂性。基于策略梯度的强化学习算法目前已成功应用于图像智能识别、机器人控制、自动驾驶路径规划等领域。然而强化学习高度依赖采样的特性决定了其训练过程需要大量样本来收敛，且决策的准确性易受到与仿真环境中不匹配的轻微干扰造成严重影响。特别是当强化学习应用于控制领域时，由于无法保证算法的收敛性，难以对其稳定性进行证明，为此，需要对强化学习进行改进。考虑到群体智能算法可通过群体协作解决复杂问题，具有自组织性及稳定性强的特征，利用其对强化学习进行优化求解是一个提高强化学习模型稳定性的有效途径。结合群体智能中的鸽群算法，对基于策略梯度的强化学习进行改进：针对求解策略梯度时存在迭代求解可能无法收敛的问题，提出了基于鸽群的强化学习算法，以最大化未来奖励为目的求解策略梯度，将鸽群算法中的适应性函数和强化学习结合估计策略的优劣，避免求解陷入死循环，提高了强化学习算法的稳定性。在具有非线性关系的两轮倒立摆机器人控制系统上进行仿真验证，实验结果表明，基于鸽群的强化学习算法能够提高系统的鲁棒性，降低计算量，减少算法对样本数据库的依赖。

Abstract

Reinforcement learning(RL) is an artificial intelligence algorithm with the advantages of clear calculation logic and easy expansion of the model.Through interacting with the environment and maximizing value functions on the premise of obtaining little or no prior information

RL can optimize the performance of strategies and effectively reduce the complexity caused by physical models .The RL algorithm based on strategy gradient has been successfully applied in many fields such as intelligent image recognition

robot control and path planning for automatic driving.However

the highly sampling-dependent characteristics of RL determine that the training process needs a large number of samples to converge

and the accuracy of decision making is easily affected by slight interference that does not match with the simulation environment.Especially when RL is applied to the control field

it is difficult to prove the stability of the algorithm because the convergence of the algorithm cannot be guaranteed.Considering that swarm intelligence algorithm can solve complex problems through group cooperation and has the characteristics of self-organization and strong stability

it is an effective way to be used for improving the stability of RL model.The pigeon-inspired optimization algorithm in swarm intelligence was combined to improve RL based on strategy gradient.A RL algorithm based on pigeon-inspired optimization was proposed to solve the strategy gradient in order to maximize long-term future rewards.Adaptive function of pigeon-inspired optimization algorithm and RL were combined to estimate the advantages and disadvantages of strategies

avoid solving into an infinite loop

and improve the stability of the algorithm.A nonlinear two-wheel inverted pendulum robot control system was selected for simulation verification.The simulation results show that the RL algorithm based on pigeon-inspired optimization can improve the robustness of the system

reduce the computational cost

and reduce the algorithm’s dependence on the sample database.

关键词

Keywords

references

PETERS J , SCHAAL S . Policy gradient methods for robotics [C ] // 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems . 2006 : 2219 - 2225 .

BAUM Y , AMICO M , HOWELL S , et al . Experimental deep reinforcement learning for error-robust gate-set design on a superconducting quantum computer [J ] . PRX Quantum , 2021 , 2 ( 4 ): 040324 .

HUA J , ZENG L , LI G , et al . Learning for a robot:deep reinforcement learning,imitation learning,transfer learning [J ] . Sensors , 2021 , 21 ( 4 ): 1278 .

SIVAK V V , EICKBUSCH A , LIU H , et al . Model-free quantum control with reinforcement learning [J ] . Physical Review X , 2022 , 12 ( 1 ): 011059 .

AGARWAL N , HAZAN E , MAJUMDAR A , et al . A regret minimization approach to iterative learning control [C ] // International Conference on Machine Learning (PMLR) . 2021 : 100 - 109 .

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks [J ] . Advances in Neural Information Processing Systems . 2012 : 1097 - 1105 .

YARATS D , FERGUS R , LAZARIC A , et al . Reinforcement learning with prototypical representations [C ] // International Conference on Machine Learning (PMLR) . 2021 : 11920 - 11931 .

DAHL G E , YU D , DENG L , et al . Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition [J ] . IEEE Transactions on Audio,Speech,and Language Processing , 2011 : 30 - 42 .

HAOXIANG W , SMYS S . Overview of configuring adaptive activation functions for deep neural networks—a comparative study [J ] . Journal of Ubiquitous Computing and Communication Technologies (UCCT) , 2021 , 3 ( 1 ): 10 - 22 .

MISHRA A , LATORRE J A , Pool J , et al . Accelerating sparse deep neural networks [J ] . arXiv preprint arXiv:2104.08378 , 2021 .

SILVER D , HUANG A , MADDISON C J , et al . Mastering the game of go with deep neural networks and tree search [J ] . Nature , 2016 ,529:484.

VINYALS O , BABUSCHKIN I , CZARNECKI W M , et al . Grandmaster level in StarCraft II using multi-agent reinforcement learning [J ] . Nature , 2019 , 575 : 350 - 354 .

HEESS N , WAYNE G , SILVER D , et al . Learning continuous control policies by stochastic value gradients [C ] // Advances in Neural Information Processing Systems . 2015 :28.

CHEN Z , CHEN B , XIE S , et al . Efficiently training on-policy actor-critic networks in robotic deep reinforcement learning with demonstration-like sampled exploration [C ] // 2021 3rd International Symposium on Robotics ＆ Intelligent Manufacturing Technology (ISRIMT) . 2021 : 292 - 298 .

WANG C , LING Y . Actor-critic tracking with precise scale estimation and advantage function [J ] . Journal of Physics Conference Series , 2021 , 1827 ( 1 ): 012064 .

ZHANG S , DUAN H . Gaussian pigeon-inspired optimization approach to orbital spacecraft formation reconfiguration [J ] . Chinese Journal of Aeronautics , 2015 , 28 ( 1 ): 200 - 205 .

ZHANG B , DUAN H . Three-dimensional path planning for uninhabited combat aerial vehicle based on predator-prey pigeon-inspired optimization in dynamic environment [J ] . IEEE/ACM Transactions on Computational Biology ＆ Bioinformatics , 2017 , 14 ( 1 ): 97 - 107 .

周雨鹏 . 基于鸽群算法的函数优化问题求解 [D ] . 长春:东北师范大学 , 2016 .

ZHOU Y P . Function optimization problem solving based on pigeon swarm algorithm [D ] . Changchun:Northeast Normal University , 2016 .

顾清华 , 孟倩倩 . 优化复杂函数的粒子群-鸽群混合优化算法 [J ] . 计算机工程与应用 , 2019 , 55 ( 22 ): 46 - 52 .

GU Q H , MENG Q Q . Hybrid particle swarm optimization and pigeon—inspired optimization algorithm for solving complex functions [J ] . Computer Engineering and Applications , 2019 , 55 ( 22 ): 46 - 52 .

胡耀龙 , 冯强 , 海星朔 , 等 . 基于自适应学习策略的改进鸽群优化算法 [J ] . 北京航空航天大学学报 , 2020 , 46 ( 12 ): 2348 - 2356 .

HU Y L , FENG Q , HAI X S , et al . Improved pigeon-inspired optimization algorithm based on adaptive learning strategy [J ] . Journal of Beijing University of Aeronautics and Astronautics , 2020 , 46 ( 12 ): 2348 - 2356 .

浏览量

427

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

带鲁棒性的子分组多重Schnorr签名方案

基于孪生网络的鲁棒性深度伪造检测方法

面向图像识别的卷积神经网络鲁棒性研究进展

面向商用活体检测平台的鲁棒性评估

基于半局部结构的异常连边识别算法