计算机扑克智能博弈研究综述

袁唯淋; 廖志勇; 高巍; 魏婷婷; 罗俊仁; 张万鹏; 陈璟

doi:10.11959/j.issn.2096-109x.2021087

您当前的位置：

首页 >

文章列表页 >

计算机扑克智能博弈研究综述

专栏Ⅱ：机器学习及安全应用 | 更新时间：2024-06-03

- 计算机扑克智能博弈研究综述
- Survey on intelligent game of computer poker
- 网络与信息安全学报 2021年7卷第5期页码：57-76
- 作者机构：
- 作者简介：
  
  [ "袁唯淋（1994− ），男，云南曲靖人，国防科技大学博士生，主要研究方向为认知决策与智能博弈、对手建模、强化学习、多智能体系统" ]
  [ "廖志勇（1995− ），男，湖南永州人，国防科技大学硕士生，主要研究方向为知识图谱、智能决策、网络通信" ]
  [ "高巍（1996− ），女，辽宁开原人，国防科技大学硕士生，主要研究方向为对手建模、意图识别、弹道规划" ]
  [ "魏婷婷（1997− ），女，内蒙古鄂尔多斯人，国防科技大学硕士生，主要研究方向为智能体建模、智能博弈、多智能体强化学习" ]
  [ "罗俊仁（1989− ），男，湖北大冶人，国防科技大学博士生，主要研究方向为智能体建模、对抗团队博弈、多智能体强化学习" ]
  [ "张万鹏（1981− ），男，四川邛崃人，国防科技大学副研究员，主要研究方向为智能决策、任务规划、自动化和控制、人机协同" ]
  [ "陈璟（1972− ），男，江西南昌人，国防科技大学教授、博士生导师，主要研究方向为人工智能、智能决策、任务规划" ]
- 基金信息：
  
  国家自然科学基金;The National Natural Science Foundation of China(61702528);国家自然科学基金;The National Natural Science Foundation of China(61806212)
- DOI：10.11959/j.issn.2096-109x.2021087
  中图分类号： TP18
- 网络首发：2021-10，
  
  纸质出版：2021-10-15
- 稿件说明：
移动端阅览
袁唯淋, 廖志勇, 高巍, 等. 计算机扑克智能博弈研究综述[J]. 网络与信息安全学报, 2021,7(5):57-76.

Weilin YUAN, Zhiyong LIAO, Wei GAO, et al. Survey on intelligent game of computer poker[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 57-76.
袁唯淋, 廖志勇, 高巍, 等. 计算机扑克智能博弈研究综述[J]. 网络与信息安全学报, 2021,7(5):57-76. DOI： 10.11959/j.issn.2096-109x.2021087.

Weilin YUAN, Zhiyong LIAO, Wei GAO, et al. Survey on intelligent game of computer poker[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 57-76. DOI： 10.11959/j.issn.2096-109x.2021087.

摘要

计算机博弈是人工智能领域的“果蝇”，备受人工智能领域研究者的关注，已然成为研究认知智能的有利平台。扑克类博弈对抗问题可建模成边界确定、规则固定的不完美信息动态博弈，计算机扑克 AI 需要具备不完全信息动态决策、对手误导欺诈行为识别以及多回合筹码和风险管理等能力。首先梳理了以德州扑克为代表的计算机扑克智能博弈的发展历程，其次针对计算机扑克智能博弈典型模型算法、关键技术以及存在的主要问题进行了综述分析，最后探讨了计算机扑克智能博弈的未来发展趋势和应用前景。

Abstract

Computer game is the drosophila in the field of artificial intelligence

which has attracted the attention of researchers in artificial intelligence

and has become an advantageous testbed for the research of cognitive intelligence.Poker game can be modeled as dynamic games with imperfect information

definite boundaries and fixed rules.Computer poker AI needs such abilities as dynamic decision-making with incomplete information

identification of misleading and fraudulent behaviors by opponents

and multi-round chips and risk management.Firstly

the development of computer poker game was introduced

which represented by Texas Hold’em poker.Then

typical intelligence game model algorithm

key techniques and existing main problems of computer poker were reviewed analysis.Finally

the future development trends and application prospect of computer intelligent poker game were discussed for cognitive intelligence.

关键词

Keywords

references

兴军亮 . 人工智能新突破：AI程序攻克多人无限注德州扑克 [EB ] .

XING J L . The new breakthrough of artificial intelligence:AI program conquered multiplayer unlimited Texas Hold 'em [EB ] .

SILVER D , HUANG AJA , MADDISON C J , et al . Mastering the game of Go with deep neural networks and tree search [J ] . Nature , 2016 , 529 ( 7587 ): 484 - 489 .

SILVER D , SCHRITTWIESER J , SIMONYAN K , et al . Mastering the game of Go without human knowledge [J ] . Nature , 2017 , 550 ( 7676 ): 354 - 359 .

BLAIR A , SAFFIDINE A . AI surpasses humans at six-player poker [J ] . Science , 2019 , 365 ( 6456 ): 864 - 865 .

MORAVCIK , MATEJ , SCHMID M , et al . DeepStack:expert-level artificial intelligence in heads-up no-limit poker [J ] . Science , 2017 :6960.

NOAM B,TUOMASS . Superhuman AI for heads-up no-limit poker:Libratus beats top professionals [J ] . Science:eaao1733 .

JIANG Q , LI K , DU B , et al . DeltaDou:expert/level doudizhu AI through self-play [C ] // Proceedings of the 28th International Joint Conference on Artificial Intelligence . 2019 1265 - 1271 .

LI J , KOYAMADA S , YE Q , et al . Suphx:mastering mahjong with deep reinforcement learning [J ] . arXiv preprint arXiv:2003.13590 , 2020 .

BERNER C , BROCKMAN G , CHAN B , et al . Dota 2 with large scale deep reinforcement learning [J ] . arXiv preprint arXiv:1912.06680 , 2019 .

YE D , LIU Z , SUN M , et al . Mastering complex control in MOBA games with deep reinforcement learning [J ] . arXiv preprint arXiv:1912.09729 , 2019 .

朱建明 , 高博 . 社交金融的信息安全风险分析与防范 [J ] . 网络与信息安全学报 , 2016 , 2 ( 3 ): 46 - 51 .

ZHU J M , GAO B . Social financial information security risk analysis and prevention [J ] . Chinese Journal of Network and Information Security , 2016 , 2 ( 3 ): 46 - 51 .

杨峻楠 , 张红旗 , 张传富 . 基于不完全信息随机博弈的防御决策方法 [J ] . 网络与信息安全学报 , 2018 , 4 ( 8 ): 12 - 20 .

YANG J N , ZHANG H Q , ZHANG C F . Defense decision-making method based on incomplete information stochastic game [J ] . Chinese Journal of Network and Information Security , 2018 , 4 ( 8 ): 12 - 20 .

PAPP D R . Dealing with imperfect information in poker [R ] . 1998 .

BILLINGS D , DAVIDSON A , SCHAEFFER J , et al . The challenge of poker [J ] . Artificial Intelligence , 2002 , 134 ( 12 ): 201 - 240 .

BILLINGS D , BURCH N , DAVIDSON A , et al . Approximating game-theoretic optimal strategies for full-scale poker [C ] // IJCAI . 2003 :661.

SCHAUENBERG T C . Opponent modelling and search in poker [D ] . Edmonton:University of Alberta , 2009 .

ZINKEVICH M , JOHANSON M , BOWLING M , et al . Regret minimization in games with incomplete information [C ] // Advances in Neural Information Processing Systems . 2008 : 1729 - 1736 .

JOHANSON M B . Robust strategies and counter-strategies:from superhuman to optimal play [D ] . Edmonton:University of Alberta , 2009 .

SCHNIZLEIN D . State translation in no-limit poker [D ] . Edmonton:University of Alberta , 2009 .

ABOU R N , SZAFRON D . Using counterfactual regret minimization to create competitive multiplayer poker agents [C ] // Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems . 2010 : 159 - 166 .

GILPIN A , SANDHOLM T , SORENSEN T B . A heads-up no-limit Texas Hold’em poker player:discretized betting models and automatically generated equilibrium-finding programs [C ] // Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems , 2008 : 911 - 918 .

BROWN N , GANZFRIED S , SANDHOLM T . Tartanian7:a champion two-player no-limit texas hold’em poker-playing program [C ] // Twenty-9 AAAI Conference on Artificial Intelligence . 2015 .

GANZFRIED S . Reflections on the first man vs.machine no-limit Texas Hold'em competition [J ] . ACM SIGecom Exchanges , 2016 , 14 ( 2 ): 2 - 15 .

BROWN N , SANDHOLM T . Baby Tartanian8:winning agent from the 2016 annual computer poker competition [C ] // IJCAI . 2016 : 4238 - 4239 .

BILLINGS D , BURCH N , DAVIDSON A , et al . Approximating game-theoretic optimal strategies for full-scale poker [C ] // Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence . 2003 : 661 - 668 .

STURTEVANT N , BOWLING M . Robust game play against unknown opponents [C ] // Proceedings of the fifth International Joint Conference on Autonomous Agents and Multiagent Systems . 2006 : 713 - 719 .

JOHANSON M B . Robust strategies and counter- strategies:building a champion level computer poker player [D ] . Edmonton:University of Alberta , 2007 .

ZINKEVICH M , JOHANSON M , BOWLING M H , et al . Regret minimization in games with incomplete information [C ] // International Conference on Neural Information Processing Systems . 2007 .

ABOU RISK N , SZAFRON D . Using counterfactual regret minimization to create competitive multiplayer poker agents [C ] // Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems . 2010 ,, 1 : 159 - 166 .

BOWLING M , BURCH N , JOHANSON M , et al . Heads-up limit hold’em poker is solved [J ] . Science , 2015 , 347 ( 6218 ): 145 - 149 .

LOCKHART E , LANCTOT M , PEROLAT J , et al . Computing approximate equilibria in sequential adversarial games by exploitability descent [J ] . arXiv preprint arXiv:1903.05614 , 2019 .

BROWN N , SANDHOLM T , AMOS B . Depth-limited solving for imperfect-information games [C ] // Advances in Neural Information Processing Systems . 2018 : 7663 - 7674 .

BROWN N , LERER A , GROSS S , et al . Deep counterfactual regret minimization [C ] // International Conference on machine learning.PMLR . 2019 : 793 - 802 .

PASSOS N M S . Poker learner:reinforcement learning applied to Texas Hold’em poker [D ] . Porto:The University of Porto , 2011 .

ZHANG L , WANG W , LI S , et al . Monte Carlo neural fictitious self-play:achieve approximate nash equilibrium of imperfect-information games [J ] . arXiv preprint arXiv:1903.09569v2 , 2019 .

ZHANG J , WANG X , YAO L , et al . Using risk dominance strategy in poker [J ] . Journal of Information Hiding and Multimedia Signal Processing , 2014 , 5 ( 3 ): 546 - 554 .

滕雯娟 . 基于虚拟遗憾最小化算法的德州扑克机器博弈研究 [D ] . 哈尔滨:哈尔滨工业大学 , 2015 .

TENG W J . Research on texas poker game based on counterfactual regret minimization algorithm [D ] . Harbin:Harbin Institute of Technology , 2015 .

代佳宁 . 基于虚拟遗憾最小化算法的非完备信息机器博弈研究 [D ] . 哈尔滨:哈尔滨工业大学 , 2017 .

DAI S N . Research on imperfect information game based on counterfactual regret minimization algorithm [D ] . Harbin :Harbin Institute of Technology , 2017 .

SKLANSKY D . The theory of poker [M ] . Two plus two publishing LLC , 1999 .

SKLANSKY D , MALMUTH M . Hold'em poker for advanced players [M ] . Two Plus Two Publishing LLC , 1999 .

JOHANSON M . Measuring the size of large no-limit poker games [J ] . arXiv preprint arXiv:1302.7008 , 2013 .

BORGHETTI B J . Opponent modeling in interesting adversarial environments [M ] . Minnesota : University of Minnesota , 2008 .

WRIGHT J R . Modeling human behavior in strategic settings [D ] . COLUMBIA:University of British Columbia , 2016 .

PLONSKY O , APEL R , ERT E , et al . Predicting human decisions with behavioral theories and machine learning [J ] . arXiv preprint arXiv:1904.06866 , 2019 .

高巍 , 罗俊仁 , 袁唯淋 , 等 . 面向对手建模的意图识别方法综述 [J ] . 网络与信息安全学报 , 2021 , 7 ( 4 ): 86 - 100 .

GAO W , LUO J R , YUAN W L , et al . A survey of intention recognition for opponent modeling [J ] . Chinese Journal of Network and Information Security , 2021 , 7 ( 4 ): 86 - 100 .

HOEHN B , SOUTHEY F , HOLTE R C , et al . Effective short-term opponent exploitation in simplified poker [C ] // AAAI . 2005 : 783 - 788 .

GANZFRIED S , SANDHOLM T . Game theory-based opponent modeling in large imperfect-information games [C ] // The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 . 2011 : 533 - 540 .

NORRIS K , WATSON I . A statistical exploitation module for Texas Hold'em:and it's benefits when used with an approximate Nash equilibrium strategy [C ] // 2013 IEEE Conference on Computational Inteligence in Games (CIG) . 2013 : 1 - 8 .

GANZFRIED S , SANDHOLM T . Safe opponent exploitation [J ] . ACM Transactions on Economics and Computation (TEAC) , 2015 , 3 ( 2 ): 1 - 28 .

SANDHOLM T . The state of solving large incomplete-information games,and application to poker [J ] . AI Magazine , 2010 , 31 ( 4 ): 13 - 32 .

SANDHOLM T . Solving imperfect-information games [J ] . Science , 2015 , 347 ( 6218 ): 122 - 123 .

SANDHOLM T , . Abstraction for solving large incomplete-information games [C ] // 29 AAAI Conference on Artificial Intelligence . 2015 .

GILPIN A , SANDHOLM T . Lossless abstraction of imperfect information games [J ] . Journal of the ACM (JACM) , 2007 , 54 ( 5 ).

GILPIN A , SANDHOLM T . Better automated abstraction techniques for imperfect information games,with application to Texas Hold’em poker [C ] // Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems . 2007 : 1 - 8 .

NOAM B , GANZFRIED S , SANDHOLM T . Hierarchical abstraction,distributed equilibrium computation,and post-processing,with application to a champion no-limit Texas Hold'em agent [C ] // Workshops at the 29 AAAI Conference on Artificial Intelligence . 2015 .

BROWN N , SANDHOLM T . Safe and nested subgame solving for imperfect-information games [C ] // Advances in Neural Information Processing Systems . 2017 : 689 - 699 .

NESTEROV . Excessive gap technique in nonsmooth convex minimization [J ] . SIAM Journal of Optimization , 2005 , 16 ( 1 ): 235 - 249 .

HART S A , MAS D , COLELL A . A simple adaptive procedure leading to correlated equilibrium [J ] . Econometrica , 2000 , 68 ( 5 ): 1127 - 1150 .

GREENWALD A , LI Z , MARKS C . Bounds for regret-matching algorithms [C ] // Proceedings of the 9 International Symposium on Artificial Intelligence and Mathematics . 2005 .

GIBSON R G , LANCTOT M , BURCH N , et al . Generalized sampling and variance in counterfactual regret minimization [C ] // Proceedings of the Twenty-Sixth Conference on Artificial Intelligence . 2012 : 3690 - 3706 .

LISY V , LANCTOT M , BOWLING M . Online monte carlo counterfactual regret minimization for search in imperfect information games [C ] // Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems . 2015 : 27 - 36 .

GANZFRIED S , SANDHOLM T . Endgame solving large imperfect-information games [C ] // Workshops at the Twenty-Ninth AAAAI Conference on Artificial Intelligence . 2015 .

BROWN N , SANDHOLM T . Solving imperfect-information games via discounted regret minimization [C ] // AAAI Conference on Artificial Intelligence (AAAI) . 2019 .

BROWN N , SANDHOLM T . Regret-based pruning in extensive form games [C ] // Advances in Neural Information Processing Systems . 2015 : 1972 - 1980 .

BROWN N , SANDHOLM T . Strategy-based warm starting for regret minimization in games [C ] // Proceedings of 30 Conference on Artificial Intelligence . 2015 : 432 - 438 .

LANCTOT M , WAUGH K , ZINKEVICH M , et al . Monte Carlo sampling for regret minimization in extensive games [C ] // Advances in Neural Information Processing Systems . 2009 : 1078 - 1086 .

LI H , HU K , GE Z , et al . Double neural counterfactual regret minimization [J ] . 2018 :

NOAM B , LERER A , GROSS S , et al . Deep counterfactual regret minimization [J ] . axXiv Preprint arXiv:1812.10607 , 2018 .

ERIC S . Single deep counterfactual regret minimization [J ] . arXiv Preprint arXiv:1901.07621 , 2018 .

FARINA G , KROER C , BROWN N , et al . Stable-predictive optimistic counterfactual regret minimization [C ] // International Conference on Machine Learning . 2019 : 1853 - 1862 .

陈学松 , 杨宜民 . 强化学习研究综述 [J ] . 计算机应用研究 , 2010 ( 8 ): 40 - 44 , 50 .

YANG X S , CHEN Y M . Reinforcement learning：survey of recent work [J ] . Application Research of Computers , 2010 ( 8 ): 40 - 44 , 50 .

HEINRICH J , SILVER D . Deep reinforcement learning from self-play in imperfect-information games [J ] . arXiv preprint arXiv:1603.01121 , 2016 .

LI K , XU H , ZHANG M , et al . OpenHoldem:an open toolkit for large-scale imperfect-information game research [J ] . arXiv preprint arXiv:2012.06168 , 2020 .

ZHA D , LAI K H , CAO Y , et al . RLCard:a toolkit for reinforcement learning in card games [J ] . arXiv preprint arXiv:1910.04376 , 2019 .

NIKOLAI Y , CAO L , RAFFEL C , et al . Poker-CNN:a pattern learning strategy for making draws and bets in poker games using convolutional networks [C ] // 30 AAAI Conference on Artificial Intelligence . 2016 .

KORB K B , NICHOLSON A , JITNAH N . Bayesian poker [J ] . arXiv preprint arXiv:1301.6711 , 2013 .

GANZFRIED S , SANDHOLM T . Endgame solving in large imperfect information games [C ] // Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems . 2015 : 37 - 45 .

SOUTHEY F , BOWLING M P , LARSON B , et al . Bayes' bluff:opponent modelling in poker [J ] . arXiv preprint arXiv:1207.1411 , 2012 .

LI H , WANG X , JIA F , et al . RLCFR:minimize counterfactual regret by deep reinforcement learning [J ] . arXiv preprint arXiv:2009.06373 , 2020 .

YANNAKAKIS G N , TOGELIUS J . Artificial intelligence and games [M ] . New York : Springer , 2018 .

浏览量

1211

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

暂无数据