强化学习中基于权重梯度下降的函数逼近方法

秦晓燕; 刘禹含; 徐云龙; 李斌

doi:10.11959/j.issn.2096-109x.2023050

您当前的位置：

首页 >

文章列表页 >

强化学习中基于权重梯度下降的函数逼近方法

学术论文 | 更新时间：2024-06-03

- 强化学习中基于权重梯度下降的函数逼近方法
- Function approximation method based on weights gradient descent in reinforcement learning
- 网络与信息安全学报 2023年9卷第4期页码：16-28
- 作者机构：
  
  1. 苏州高博软件技术职业学院信息与软件学院，江苏苏州 215163
  2. 滑铁卢大学，安大略滑铁卢 N2L3G4
  3. 苏州大学应用技术学院，江苏苏州 215325
  4. 苏州大学计算机科学与技术学院，江苏苏州 215325
- 作者简介：
  
  [ "秦晓燕（1984- ），女，江苏泰州人，苏州高博软件技术职业学院副教授，主要研究方向为软件工程、人工智能" ]
  [ "刘禹含（2000- ），女，黑龙江大庆人，加拿大滑铁卢大学硕士生，主要研究方向为数字媒体技术" ]
  [ "徐云龙（1964- ），男，江苏苏州人，苏州大学副教授，主要研究方向为机器学习、操作系统" ]
  [ "李斌（1994- ），男，江苏镇江人，主要研究方向为强化学习" ]
- 基金信息：
  
  国家自然科学基金;The National Natural Science Foundation of China(61772355);国家自然科学基金;The National Natural Science Foundation of China(61702055);国家自然科学基金;The National Natural Science Foundation of China(61876217);国家自然科学基金;The National Natural Science Foundation of China(62176175);江苏省高等学校自然科学研究重大项目;Jiangsu Province Natural Science Research University Major Projects(18KJA520011);江苏省高等学校自然科学研究重大项目;Jiangsu Province Natural Science Research University Major Projects(17KJA520004);苏州市应用基础研究计划工业部分;Suzhou Industrial Application of Basic Research Program Part(SYG201422);江苏省高职院校教师专业带头人高端研修项目;Jiangsu Province High End Research and Training Project for Professional Leaders of Teachers in Vocational Colleges(2021GRFX052);江苏高校优势学科建设工程资助项目;Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions;江苏省职业教育软件技术“双师型”名师工作室资助项目;Project Funded byJiangsu Province Vocational Education the Double-Qualified of Teaching Stu-dio in Software Technology
- DOI：10.11959/j.issn.2096-109x.2023050
  中图分类号： TP18
- 网络首发：2023-08，
  
  纸质出版：2023-08-25
- 稿件说明：
移动端阅览
秦晓燕, 刘禹含, 徐云龙, 等. 强化学习中基于权重梯度下降的函数逼近方法[J]. 网络与信息安全学报, 2023,9(4):16-28.

Xiaoyan QIN, Yuhan LIU, Yunlong XU, et al. Function approximation method based on weights gradient descent in reinforcement learning[J]. Chinese Journal of Network and Information Security, 2023, 9(4): 16-28.
秦晓燕, 刘禹含, 徐云龙, 等. 强化学习中基于权重梯度下降的函数逼近方法[J]. 网络与信息安全学报, 2023,9(4):16-28. DOI： 10.11959/j.issn.2096-109x.2023050.

Xiaoyan QIN, Yuhan LIU, Yunlong XU, et al. Function approximation method based on weights gradient descent in reinforcement learning[J]. Chinese Journal of Network and Information Security, 2023, 9(4): 16-28. DOI： 10.11959/j.issn.2096-109x.2023050.

摘要

函数逼近法（function approximation）是强化学习领域中的一个研究热点，可以有效处理强化学习中大规模、连续状态和动作空间的问题。基于梯度下降（gradient descent）的函数逼近方法虽然是强化学习中使用最广泛的方法之一，但该算法对步长参数的要求较高，取值不当易产生收敛速度慢、收敛不稳定甚至发散的情况。针对这类问题，通过围绕基于函数逼近的TD（TD，temporal difference）算法，在最小二乘方法和梯度下降方法的基础上对权重的更新方法进行了改进，利用最小二乘方法处理值函数求解权重值，并结合时序差分和梯度下降的思想求出权重之间的误差，并利用该误差直接更新权重，从而提出一种权重梯度下降（WGD，weight gradient descent）方法。该方法以全新的方式更新权重，有效降低算法对计算资源的消耗，并且可以有效地对其他基于梯度下降的函数逼近算法进行改进，广泛应用于诸多基于梯度下降的强化学习算法。实验表明，WGD 方法能够在更广泛的空间中调整参数，可以有效降低算法发散的可能性，在保证算法拥有良好收敛效果的同时，提高算法的收敛速度。

Abstract

Function approximation has gained significant attention in reinforcement learning research as it effectively addresses problems with large-scale

continuous state

and action space.Although the function approximation algorithm based on gradient descent method is one of the most widely used methods in reinforcement learning

it requires careful tuning of the step size parameter as an inappropriate value can lead to slow convergence

unstable convergence

or even divergence.To address these issues

an improvement was made around the temporal-difference (TD) algorithm based on function approximation.The weight update method was enhanced using both the least squares method and gradient descent

resulting in the proposed weights gradient descent (WGD) method.The least squares were used to calculate the weights

combining the ideas of TD and gradient descent to find the error between the weights.And this error was used to directly update the weights.By this method

the weights were updated in a new manner

effectively reducing the consumption of computing resources by the algorithm enhancing other gradient descent-based function approximation algorithms.The WGD method is widely applicable in various gradient descent-based reinforcement learning algorithms.The results show that WGD method can adjust parameters within a wider space

effectively reducing the possibility of algorithm divergence.Additionally

it achieves better performance while improving the convergence speed of the algorithm.

关键词

Keywords

references

SUTTON R S , BARTO A G . Reinforcement learning:an introduction [J ] . IEEE Transactions on Neural Networks , 1998 , 9 ( 5 ): 1054 - 1054 .

陈兴国 , 俞扬 . 强化学习及其在电脑围棋中的应用 [J ] . 自动化学报 , 2016 , 42 ( 5 ): 685 - 695 .

CHEN X G , YU Y . Reinforcement learning and its application to the game of GO [J ] . Acta Automatica Sinica , 2016 , 42 ( 5 ): 685 - 695 .

HU Y J , GAO Y , AN B . Multiagent reinforcement learning with unshared value functions [J ] . IEEE Transactions on Cybernetics , 2015 , 45 ( 4 ): 647 - 662 .

POLYDOROS A S , NALPANTIDIS L . Survey of model-based reinforcement learning:applications on robotics [J ] . Journal of Intelligent ＆ Robotic Systems , 2017 , 86 ( 2 ): 1 - 21 .

SSENGONZI C , KOGEDA O P , OLWAL T O . A survey of deep reinforcement learning application in 5G and beyond network slicing and virtualization [R ] . 2022 .

WU Y , MOZIFIAN M , SHKURTI F . Shaping rewards for reinforcement learning with imperfect demonstrations using generative models [C ] // 2021 IEEE International Conference on Robotics and Automation (ICRA) . 2021 : 6628 - 6634 .

XU XIN , ZUO LEI , HUANG Z H . Reinforcement learning algorithms with function approximation:Recent advances and applications [J ] . Information Sciences , 2014 , 261 ( 5 ): 1 - 31 .

GRONDMAN I , BUSONIU L , LOPES G A D , et al . A survey of actor-critic reinforcement learning:standard and natural policy gradients [J ] . IEEE Transactions on Systems,Man,and Cybernetics,Part C (Applications and Reviews) , 2012 , 42 ( 6 ): 1291 - 1307 .

VAN-SEIJEN H , MAHMOOD A R , PILARSKI P M , et al . True online temporal-difference learning [J ] . Journal of Machine Learning Research , 2015 , 17 ( 1 ): 5057 - 5096 .

LI K , BURDICK J W . A function approximation method for model-based high-dimensional inverse reinforcement learning [R ] . 2017 .

FUJIMOTO S , GU S S . A minimalist approach to offline reinforcement learning [C ] // Advances in Neural Information Processing Systems , 2021 , 34 : 20132 - 20145 .

THOMAS P S , BRUNSKILL E . Policy gradient methods for reinforcement learning with function approximation and action-dependent baselines [R ] . 2017 .

MNIH V , KAVUKCUOGLU K , SILVER D , et al . Human-level control through deep reinforcement learning [J ] . Nature , 2015 , 518 ( 7540 ): 529 - 533 .

郭潇逍 , 李程 , 梅俏竹 . 深度学习在游戏中的应用 [J ] . 自动化学报 , 2016 , 42 ( 5 ): 676 - 684 .

GUO X X , LI C , MEI Q Z . Deep learning applied to games [J ] . Acta Automatica Sinica , 2016 , 42 ( 5 ): 676 - 684 .

GEIST M , PIETQUIN O . Parametric value function approximation:a unified view [C ] // Proceedings of the 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning . 2011 .

FAIRBANK M , ALONSO E . The divergence of reinforcement learning algorithms with value-iteration and function approximation [C ] // International Joint Conference on Neural Networks . 2012 .

BHANDARI J , RUSSO D , SINGAL R . A finite time analysis of temporal difference learning with linear function approximation [R ] . 2018 .

AWHEDA M D , SCHWARTZ H M . A residual gradient fuzzy reinforcement learning algorithm for differential games [J ] . International Journal of Fuzzy Systems , 2017 .

BO L , JI L , GHAVAMZADEH M , et al . Proximal gradient temporal difference learning algorithms [C ] // International Joint Conference on Artificial Intelligence . 2016 .

浏览量

125

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

语义通信下基于强化学习的无人机通信容错协同控制

基于攻击图和深度Q学习网络的自动化安全分析与渗透测试模型

基于强化学习的自动化Windows域渗透方法

基于强化学习的低时延车联网群密钥分配管理技术