Robust reinforcement learning algorithm based on pigeon-inspired optimization

Mingying ZHANG; Bing HUA; Yuguang ZHANG; Haidong LI; Mohong ZHENG

doi:10.11959/j.issn.2096-109x.2022064

您当前的位置：

首页 >

文章列表页 >

Robust reinforcement learning algorithm based on pigeon-inspired optimization

Topic: Big Data and Artifical Intelligence Security | 更新时间：2024-06-03

- Robust reinforcement learning algorithm based on pigeon-inspired optimization
- Chinese Journal of Network and Information Security Vol. 8, Issue 5, Pages: 66-74(2022)
- 作者机构：
  
  1. 中国电子技术标准化研究院，北京 100007
  2. 南京航空航天大学航天学院，江苏南京 211106
  3. 中国电子科技集团公司第七研究所，广东广州 510000
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2022064
  CLC： TP393
- Online First：2022-10，
  
  Published：15 October 2022
- 稿件说明：
移动端阅览
Mingying ZHANG, Bing HUA, Yuguang ZHANG, et al. Robust reinforcement learning algorithm based on pigeon-inspired optimization[J]. Chinese Journal of Network and Information Security, 2022, 8(5): 66-74.
DOI：

Mingying ZHANG, Bing HUA, Yuguang ZHANG, et al. Robust reinforcement learning algorithm based on pigeon-inspired optimization[J]. Chinese Journal of Network and Information Security, 2022, 8(5): 66-74. DOI： 10.11959/j.issn.2096-109x.2022064.

摘要

强化学习是一种人工智能算法，具有计算逻辑清晰、模型易扩展的优点，可以在较少甚至没有先验信息的前提下，通过和环境交互并最大化值函数，调优策略性能，有效地降低物理模型引起的复杂性。基于策略梯度的强化学习算法目前已成功应用于图像智能识别、机器人控制、自动驾驶路径规划等领域。然而强化学习高度依赖采样的特性决定了其训练过程需要大量样本来收敛，且决策的准确性易受到与仿真环境中不匹配的轻微干扰造成严重影响。特别是当强化学习应用于控制领域时，由于无法保证算法的收敛性，难以对其稳定性进行证明，为此，需要对强化学习进行改进。考虑到群体智能算法可通过群体协作解决复杂问题，具有自组织性及稳定性强的特征，利用其对强化学习进行优化求解是一个提高强化学习模型稳定性的有效途径。结合群体智能中的鸽群算法，对基于策略梯度的强化学习进行改进：针对求解策略梯度时存在迭代求解可能无法收敛的问题，提出了基于鸽群的强化学习算法，以最大化未来奖励为目的求解策略梯度，将鸽群算法中的适应性函数和强化学习结合估计策略的优劣，避免求解陷入死循环，提高了强化学习算法的稳定性。在具有非线性关系的两轮倒立摆机器人控制系统上进行仿真验证，实验结果表明，基于鸽群的强化学习算法能够提高系统的鲁棒性，降低计算量，减少算法对样本数据库的依赖。

Abstract

Reinforcement learning(RL) is an artificial intelligence algorithm with the advantages of clear calculation logic and easy expansion of the model.Through interacting with the environment and maximizing value functions on the premise of obtaining little or no prior information

RL can optimize the performance of strategies and effectively reduce the complexity caused by physical models .The RL algorithm based on strategy gradient has been successfully applied in many fields such as intelligent image recognition

robot control and path planning for automatic driving.However

the highly sampling-dependent characteristics of RL determine that the training process needs a large number of samples to converge

and the accuracy of decision making is easily affected by slight interference that does not match with the simulation environment.Especially when RL is applied to the control field

it is difficult to prove the stability of the algorithm because the convergence of the algorithm cannot be guaranteed.Considering that swarm intelligence algorithm can solve complex problems through group cooperation and has the characteristics of self-organization and strong stability

it is an effective way to be used for improving the stability of RL model.The pigeon-inspired optimization algorithm in swarm intelligence was combined to improve RL based on strategy gradient.A RL algorithm based on pigeon-inspired optimization was proposed to solve the strategy gradient in order to maximize long-term future rewards.Adaptive function of pigeon-inspired optimization algorithm and RL were combined to estimate the advantages and disadvantages of strategies

avoid solving into an infinite loop

and improve the stability of the algorithm.A nonlinear two-wheel inverted pendulum robot control system was selected for simulation verification.The simulation results show that the RL algorithm based on pigeon-inspired optimization can improve the robustness of the system

reduce the computational cost

and reduce the algorithm’s dependence on the sample database.

关键词

Keywords

references

PETERS J , SCHAAL S . Policy gradient methods for robotics [C ] // 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems . 2006 : 2219 - 2225 .

BAUM Y , AMICO M , HOWELL S , et al . Experimental deep reinforcement learning for error-robust gate-set design on a superconducting quantum computer [J ] . PRX Quantum , 2021 , 2 ( 4 ): 040324 .

HUA J , ZENG L , LI G , et al . Learning for a robot:deep reinforcement learning,imitation learning,transfer learning [J ] . Sensors , 2021 , 21 ( 4 ): 1278 .

SIVAK V V , EICKBUSCH A , LIU H , et al . Model-free quantum control with reinforcement learning [J ] . Physical Review X , 2022 , 12 ( 1 ): 011059 .

AGARWAL N , HAZAN E , MAJUMDAR A , et al . A regret minimization approach to iterative learning control [C ] // International Conference on Machine Learning (PMLR) . 2021 : 100 - 109 .

KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks [J ] . Advances in Neural Information Processing Systems . 2012 : 1097 - 1105 .

YARATS D , FERGUS R , LAZARIC A , et al . Reinforcement learning with prototypical representations [C ] // International Conference on Machine Learning (PMLR) . 2021 : 11920 - 11931 .

DAHL G E , YU D , DENG L , et al . Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition [J ] . IEEE Transactions on Audio,Speech,and Language Processing , 2011 : 30 - 42 .

HAOXIANG W , SMYS S . Overview of configuring adaptive activation functions for deep neural networks—a comparative study [J ] . Journal of Ubiquitous Computing and Communication Technologies (UCCT) , 2021 , 3 ( 1 ): 10 - 22 .

MISHRA A , LATORRE J A , Pool J , et al . Accelerating sparse deep neural networks [J ] . arXiv preprint arXiv:2104.08378 , 2021 .

SILVER D , HUANG A , MADDISON C J , et al . Mastering the game of go with deep neural networks and tree search [J ] . Nature , 2016 ,529:484.

VINYALS O , BABUSCHKIN I , CZARNECKI W M , et al . Grandmaster level in StarCraft II using multi-agent reinforcement learning [J ] . Nature , 2019 , 575 : 350 - 354 .

HEESS N , WAYNE G , SILVER D , et al . Learning continuous control policies by stochastic value gradients [C ] // Advances in Neural Information Processing Systems . 2015 :28.

CHEN Z , CHEN B , XIE S , et al . Efficiently training on-policy actor-critic networks in robotic deep reinforcement learning with demonstration-like sampled exploration [C ] // 2021 3rd International Symposium on Robotics ＆ Intelligent Manufacturing Technology (ISRIMT) . 2021 : 292 - 298 .

WANG C , LING Y . Actor-critic tracking with precise scale estimation and advantage function [J ] . Journal of Physics Conference Series , 2021 , 1827 ( 1 ): 012064 .

ZHANG S , DUAN H . Gaussian pigeon-inspired optimization approach to orbital spacecraft formation reconfiguration [J ] . Chinese Journal of Aeronautics , 2015 , 28 ( 1 ): 200 - 205 .

ZHANG B , DUAN H . Three-dimensional path planning for uninhabited combat aerial vehicle based on predator-prey pigeon-inspired optimization in dynamic environment [J ] . IEEE/ACM Transactions on Computational Biology ＆ Bioinformatics , 2017 , 14 ( 1 ): 97 - 107 .

周雨鹏 . 基于鸽群算法的函数优化问题求解 [D ] . 长春:东北师范大学 , 2016 .

ZHOU Y P . Function optimization problem solving based on pigeon swarm algorithm [D ] . Changchun:Northeast Normal University , 2016 .

顾清华 , 孟倩倩 . 优化复杂函数的粒子群-鸽群混合优化算法 [J ] . 计算机工程与应用 , 2019 , 55 ( 22 ): 46 - 52 .

GU Q H , MENG Q Q . Hybrid particle swarm optimization and pigeon—inspired optimization algorithm for solving complex functions [J ] . Computer Engineering and Applications , 2019 , 55 ( 22 ): 46 - 52 .

胡耀龙 , 冯强 , 海星朔 , 等 . 基于自适应学习策略的改进鸽群优化算法 [J ] . 北京航空航天大学学报 , 2020 , 46 ( 12 ): 2348 - 2356 .

HU Y L , FENG Q , HAI X S , et al . Improved pigeon-inspired optimization algorithm based on adaptive learning strategy [J ] . Journal of Beijing University of Aeronautics and Astronautics , 2020 , 46 ( 12 ): 2348 - 2356 .

Views

427

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Robust Schnorr-based subgroup multi-signature scheme

Robust deepfake detection method based on siamese network

Research on the robustness of convolutional neural networks in image recognition

Robustness evaluation of commercial liveness detection platform

Abnormal link detection algorithm based on semi-local structure

Related Author

WANG Zhiwei

ZHU Qiuchi

ZHANG Zhenqi

LIN Shanhe

Ping YI

Li PAN

Dian LIN

Ling PANG

Related Institution

School of Computer Science, Nanjing University of Posts and Telecommunications

Jiangsu Key Laboratory of Big Data Security and Intelligent Processing, Nanjing University of Posts and Telecommunications

Fujian Star-net Communication Company Limited

The College of Information Engineering, Zhejiang University of Technology

National Key Laboratory of Science and Technology on Information System Security

⁰