Function approximation method based on weights gradient descent in reinforcement learning

Xiaoyan QIN; Yuhan LIU; Yunlong XU; Bin LI

doi:10.11959/j.issn.2096-109x.2023050

您当前的位置：

首页 >

文章列表页 >

Function approximation method based on weights gradient descent in reinforcement learning

Papers | 更新时间：2024-06-03

- Function approximation method based on weights gradient descent in reinforcement learning
- Chinese Journal of Network and Information Security Vol. 9, Issue 4, Pages: 16-28(2023)
- 作者机构：
  
  1. 苏州高博软件技术职业学院信息与软件学院，江苏苏州 215163
  2. 滑铁卢大学，安大略滑铁卢 N2L3G4
  3. 苏州大学应用技术学院，江苏苏州 215325
  4. 苏州大学计算机科学与技术学院，江苏苏州 215325
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2023050
  CLC： TP18
- Online First：2023-08，
  
  Published：25 August 2023
- 稿件说明：
移动端阅览
Xiaoyan QIN, Yuhan LIU, Yunlong XU, et al. Function approximation method based on weights gradient descent in reinforcement learning[J]. Chinese Journal of Network and Information Security, 2023, 9(4): 16-28.
DOI：

Xiaoyan QIN, Yuhan LIU, Yunlong XU, et al. Function approximation method based on weights gradient descent in reinforcement learning[J]. Chinese Journal of Network and Information Security, 2023, 9(4): 16-28. DOI： 10.11959/j.issn.2096-109x.2023050.

摘要

函数逼近法（function approximation）是强化学习领域中的一个研究热点，可以有效处理强化学习中大规模、连续状态和动作空间的问题。基于梯度下降（gradient descent）的函数逼近方法虽然是强化学习中使用最广泛的方法之一，但该算法对步长参数的要求较高，取值不当易产生收敛速度慢、收敛不稳定甚至发散的情况。针对这类问题，通过围绕基于函数逼近的TD（TD，temporal difference）算法，在最小二乘方法和梯度下降方法的基础上对权重的更新方法进行了改进，利用最小二乘方法处理值函数求解权重值，并结合时序差分和梯度下降的思想求出权重之间的误差，并利用该误差直接更新权重，从而提出一种权重梯度下降（WGD，weight gradient descent）方法。该方法以全新的方式更新权重，有效降低算法对计算资源的消耗，并且可以有效地对其他基于梯度下降的函数逼近算法进行改进，广泛应用于诸多基于梯度下降的强化学习算法。实验表明，WGD 方法能够在更广泛的空间中调整参数，可以有效降低算法发散的可能性，在保证算法拥有良好收敛效果的同时，提高算法的收敛速度。

Abstract

Function approximation has gained significant attention in reinforcement learning research as it effectively addresses problems with large-scale

continuous state

and action space.Although the function approximation algorithm based on gradient descent method is one of the most widely used methods in reinforcement learning

it requires careful tuning of the step size parameter as an inappropriate value can lead to slow convergence

unstable convergence

or even divergence.To address these issues

an improvement was made around the temporal-difference (TD) algorithm based on function approximation.The weight update method was enhanced using both the least squares method and gradient descent

resulting in the proposed weights gradient descent (WGD) method.The least squares were used to calculate the weights

combining the ideas of TD and gradient descent to find the error between the weights.And this error was used to directly update the weights.By this method

the weights were updated in a new manner

effectively reducing the consumption of computing resources by the algorithm enhancing other gradient descent-based function approximation algorithms.The WGD method is widely applicable in various gradient descent-based reinforcement learning algorithms.The results show that WGD method can adjust parameters within a wider space

effectively reducing the possibility of algorithm divergence.Additionally

it achieves better performance while improving the convergence speed of the algorithm.

关键词

Keywords

references

SUTTON R S , BARTO A G . Reinforcement learning:an introduction [J ] . IEEE Transactions on Neural Networks , 1998 , 9 ( 5 ): 1054 - 1054 .

陈兴国 , 俞扬 . 强化学习及其在电脑围棋中的应用 [J ] . 自动化学报 , 2016 , 42 ( 5 ): 685 - 695 .

CHEN X G , YU Y . Reinforcement learning and its application to the game of GO [J ] . Acta Automatica Sinica , 2016 , 42 ( 5 ): 685 - 695 .

HU Y J , GAO Y , AN B . Multiagent reinforcement learning with unshared value functions [J ] . IEEE Transactions on Cybernetics , 2015 , 45 ( 4 ): 647 - 662 .

POLYDOROS A S , NALPANTIDIS L . Survey of model-based reinforcement learning:applications on robotics [J ] . Journal of Intelligent ＆ Robotic Systems , 2017 , 86 ( 2 ): 1 - 21 .

SSENGONZI C , KOGEDA O P , OLWAL T O . A survey of deep reinforcement learning application in 5G and beyond network slicing and virtualization [R ] . 2022 .

WU Y , MOZIFIAN M , SHKURTI F . Shaping rewards for reinforcement learning with imperfect demonstrations using generative models [C ] // 2021 IEEE International Conference on Robotics and Automation (ICRA) . 2021 : 6628 - 6634 .

XU XIN , ZUO LEI , HUANG Z H . Reinforcement learning algorithms with function approximation:Recent advances and applications [J ] . Information Sciences , 2014 , 261 ( 5 ): 1 - 31 .

GRONDMAN I , BUSONIU L , LOPES G A D , et al . A survey of actor-critic reinforcement learning:standard and natural policy gradients [J ] . IEEE Transactions on Systems,Man,and Cybernetics,Part C (Applications and Reviews) , 2012 , 42 ( 6 ): 1291 - 1307 .

VAN-SEIJEN H , MAHMOOD A R , PILARSKI P M , et al . True online temporal-difference learning [J ] . Journal of Machine Learning Research , 2015 , 17 ( 1 ): 5057 - 5096 .

LI K , BURDICK J W . A function approximation method for model-based high-dimensional inverse reinforcement learning [R ] . 2017 .

FUJIMOTO S , GU S S . A minimalist approach to offline reinforcement learning [C ] // Advances in Neural Information Processing Systems , 2021 , 34 : 20132 - 20145 .

THOMAS P S , BRUNSKILL E . Policy gradient methods for reinforcement learning with function approximation and action-dependent baselines [R ] . 2017 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

郭潇逍 , 李程 , 梅俏竹 . 深度学习在游戏中的应用 [J ] . 自动化学报 , 2016 , 42 ( 5 ): 676 - 684 .

GUO X X , LI C , MEI Q Z . Deep learning applied to games [J ] . Acta Automatica Sinica , 2016 , 42 ( 5 ): 676 - 684 .

GEIST M , PIETQUIN O . Parametric value function approximation:a unified view [C ] // Proceedings of the 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning . 2011 .

FAIRBANK M , ALONSO E . The divergence of reinforcement learning algorithms with value-iteration and function approximation [C ] // International Joint Conference on Neural Networks . 2012 .

BHANDARI J , RUSSO D , SINGAL R . A finite time analysis of temporal difference learning with linear function approximation [R ] . 2018 .

AWHEDA M D , SCHWARTZ H M . A residual gradient fuzzy reinforcement learning algorithm for differential games [J ] . International Journal of Fuzzy Systems , 2017 .

BO L , JI L , GHAVAMZADEH M , et al . Proximal gradient temporal difference learning algorithms [C ] // International Joint Conference on Artificial Intelligence . 2016 .

Views

125

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Semantic communication aware reinforcement learning for communication fault-tolerant UAV collaborative control

Autonomous security analysis and penetration testing model based on attack graph and deep Q-learning network

Automated Windows domain penetration method based on reinforcement learning

Reinforcement learning based group key agreement scheme with reduced latency for VANET

Related Author

WANG Ran

FENG Bohao

GU Hongyu

ZHANG Yang

Zhanhua ZHANG

Taojie DING

Guoqing HU

Cheng FAN

Related Institution

College of Computer Science and Technology Nanjing University of Aeronautics and Astronautics

School of Computer Science and Artificial Intelligence Wuhan University of Technology

College of Computer, Nanjing University of Posts and Telecommunications

Jiangsu Provincial Key Laboratory of Wireless Sensor Network High Technology Research

School of Informatics,Xiamen University

⁰