Survey on intelligent game of computer poker

Weilin YUAN; Zhiyong LIAO; Wei GAO; Tingting WEI; Junren LUO; Wanpeng ZAHNG; Jing CHEN

doi:10.11959/j.issn.2096-109x.2021087

您当前的位置：

首页 >

文章列表页 >

Survey on intelligent game of computer poker

TopicⅡ: Machine Learning and Security Application | 更新时间：2024-06-03

- Survey on intelligent game of computer poker
- Chinese Journal of Network and Information Security Vol. 7, Issue 5, Pages: 57-76(2021)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2021087
  CLC： TP18
- Online First：2021-10，
  
  Published：15 October 2021
- 稿件说明：
移动端阅览
Weilin YUAN, Zhiyong LIAO, Wei GAO, et al. Survey on intelligent game of computer poker[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 57-76.
DOI：

Weilin YUAN, Zhiyong LIAO, Wei GAO, et al. Survey on intelligent game of computer poker[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 57-76. DOI： 10.11959/j.issn.2096-109x.2021087.

摘要

计算机博弈是人工智能领域的“果蝇”，备受人工智能领域研究者的关注，已然成为研究认知智能的有利平台。扑克类博弈对抗问题可建模成边界确定、规则固定的不完美信息动态博弈，计算机扑克 AI 需要具备不完全信息动态决策、对手误导欺诈行为识别以及多回合筹码和风险管理等能力。首先梳理了以德州扑克为代表的计算机扑克智能博弈的发展历程，其次针对计算机扑克智能博弈典型模型算法、关键技术以及存在的主要问题进行了综述分析，最后探讨了计算机扑克智能博弈的未来发展趋势和应用前景。

Abstract

Computer game is the drosophila in the field of artificial intelligence

which has attracted the attention of researchers in artificial intelligence

and has become an advantageous testbed for the research of cognitive intelligence.Poker game can be modeled as dynamic games with imperfect information

definite boundaries and fixed rules.Computer poker AI needs such abilities as dynamic decision-making with incomplete information

identification of misleading and fraudulent behaviors by opponents

and multi-round chips and risk management.Firstly

the development of computer poker game was introduced

which represented by Texas Hold’em poker.Then

typical intelligence game model algorithm

key techniques and existing main problems of computer poker were reviewed analysis.Finally

the future development trends and application prospect of computer intelligent poker game were discussed for cognitive intelligence.

关键词

Keywords

references

兴军亮 . 人工智能新突破：AI程序攻克多人无限注德州扑克 [EB ] .

XING J L . The new breakthrough of artificial intelligence:AI program conquered multiplayer unlimited Texas Hold 'em [EB ] .

SILVER D , HUANG AJA , MADDISON C J , et al . Mastering the game of Go with deep neural networks and tree search [J ] . Nature , 2016 , 529 ( 7587 ): 484 - 489 .

SILVER D , SCHRITTWIESER J , SIMONYAN K , et al . Mastering the game of Go without human knowledge [J ] . Nature , 2017 , 550 ( 7676 ): 354 - 359 .

BLAIR A , SAFFIDINE A . AI surpasses humans at six-player poker [J ] . Science , 2019 , 365 ( 6456 ): 864 - 865 .

MORAVCIK , MATEJ , SCHMID M , et al . DeepStack:expert-level artificial intelligence in heads-up no-limit poker [J ] . Science , 2017 :6960.

NOAM B,TUOMASS . Superhuman AI for heads-up no-limit poker:Libratus beats top professionals [J ] . Science:eaao1733 .

JIANG Q , LI K , DU B , et al . DeltaDou:expert/level doudizhu AI through self-play [C ] // Proceedings of the 28th International Joint Conference on Artificial Intelligence . 2019 1265 - 1271 .

LI J , KOYAMADA S , YE Q , et al . Suphx:mastering mahjong with deep reinforcement learning [J ] . arXiv preprint arXiv:2003.13590 , 2020 .

BERNER C , BROCKMAN G , CHAN B , et al . Dota 2 with large scale deep reinforcement learning [J ] . arXiv preprint arXiv:1912.06680 , 2019 .

YE D , LIU Z , SUN M , et al . Mastering complex control in MOBA games with deep reinforcement learning [J ] . arXiv preprint arXiv:1912.09729 , 2019 .

朱建明 , 高博 . 社交金融的信息安全风险分析与防范 [J ] . 网络与信息安全学报 , 2016 , 2 ( 3 ): 46 - 51 .

ZHU J M , GAO B . Social financial information security risk analysis and prevention [J ] . Chinese Journal of Network and Information Security , 2016 , 2 ( 3 ): 46 - 51 .

杨峻楠 , 张红旗 , 张传富 . 基于不完全信息随机博弈的防御决策方法 [J ] . 网络与信息安全学报 , 2018 , 4 ( 8 ): 12 - 20 .

YANG J N , ZHANG H Q , ZHANG C F . Defense decision-making method based on incomplete information stochastic game [J ] . Chinese Journal of Network and Information Security , 2018 , 4 ( 8 ): 12 - 20 .

PAPP D R . Dealing with imperfect information in poker [R ] . 1998 .

BILLINGS D , DAVIDSON A , SCHAEFFER J , et al . The challenge of poker [J ] . Artificial Intelligence , 2002 , 134 ( 12 ): 201 - 240 .

BILLINGS D , BURCH N , DAVIDSON A , et al . Approximating game-theoretic optimal strategies for full-scale poker [C ] // IJCAI . 2003 :661.

SCHAUENBERG T C . Opponent modelling and search in poker [D ] . Edmonton:University of Alberta , 2009 .

ZINKEVICH M , JOHANSON M , BOWLING M , et al . Regret minimization in games with incomplete information [C ] // Advances in Neural Information Processing Systems . 2008 : 1729 - 1736 .

JOHANSON M B . Robust strategies and counter-strategies:from superhuman to optimal play [D ] . Edmonton:University of Alberta , 2009 .

SCHNIZLEIN D . State translation in no-limit poker [D ] . Edmonton:University of Alberta , 2009 .

ABOU R N , SZAFRON D . Using counterfactual regret minimization to create competitive multiplayer poker agents [C ] // Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems . 2010 : 159 - 166 .

GILPIN A , SANDHOLM T , SORENSEN T B . A heads-up no-limit Texas Hold’em poker player:discretized betting models and automatically generated equilibrium-finding programs [C ] // Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems , 2008 : 911 - 918 .

BROWN N , GANZFRIED S , SANDHOLM T . Tartanian7:a champion two-player no-limit texas hold’em poker-playing program [C ] // Twenty-9 AAAI Conference on Artificial Intelligence . 2015 .

GANZFRIED S . Reflections on the first man vs.machine no-limit Texas Hold'em competition [J ] . ACM SIGecom Exchanges , 2016 , 14 ( 2 ): 2 - 15 .

BROWN N , SANDHOLM T . Baby Tartanian8:winning agent from the 2016 annual computer poker competition [C ] // IJCAI . 2016 : 4238 - 4239 .

BILLINGS D , BURCH N , DAVIDSON A , et al . Approximating game-theoretic optimal strategies for full-scale poker [C ] // Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence . 2003 : 661 - 668 .

STURTEVANT N , BOWLING M . Robust game play against unknown opponents [C ] // Proceedings of the fifth International Joint Conference on Autonomous Agents and Multiagent Systems . 2006 : 713 - 719 .

JOHANSON M B . Robust strategies and counter- strategies:building a champion level computer poker player [D ] . Edmonton:University of Alberta , 2007 .

ZINKEVICH M , JOHANSON M , BOWLING M H , et al . Regret minimization in games with incomplete information [C ] // International Conference on Neural Information Processing Systems . 2007 .

ABOU RISK N , SZAFRON D . Using counterfactual regret minimization to create competitive multiplayer poker agents [C ] // Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems . 2010 ,, 1 : 159 - 166 .

BOWLING M , BURCH N , JOHANSON M , et al . Heads-up limit hold’em poker is solved [J ] . Science , 2015 , 347 ( 6218 ): 145 - 149 .

LOCKHART E , LANCTOT M , PEROLAT J , et al . Computing approximate equilibria in sequential adversarial games by exploitability descent [J ] . arXiv preprint arXiv:1903.05614 , 2019 .

BROWN N , SANDHOLM T , AMOS B . Depth-limited solving for imperfect-information games [C ] // Advances in Neural Information Processing Systems . 2018 : 7663 - 7674 .

BROWN N , LERER A , GROSS S , et al . Deep counterfactual regret minimization [C ] // International Conference on machine learning.PMLR . 2019 : 793 - 802 .

PASSOS N M S . Poker learner:reinforcement learning applied to Texas Hold’em poker [D ] . Porto:The University of Porto , 2011 .

ZHANG L , WANG W , LI S , et al . Monte Carlo neural fictitious self-play:achieve approximate nash equilibrium of imperfect-information games [J ] . arXiv preprint arXiv:1903.09569v2 , 2019 .

ZHANG J , WANG X , YAO L , et al . Using risk dominance strategy in poker [J ] . Journal of Information Hiding and Multimedia Signal Processing , 2014 , 5 ( 3 ): 546 - 554 .

滕雯娟 . 基于虚拟遗憾最小化算法的德州扑克机器博弈研究 [D ] . 哈尔滨:哈尔滨工业大学 , 2015 .

TENG W J . Research on texas poker game based on counterfactual regret minimization algorithm [D ] . Harbin:Harbin Institute of Technology , 2015 .

代佳宁 . 基于虚拟遗憾最小化算法的非完备信息机器博弈研究 [D ] . 哈尔滨:哈尔滨工业大学 , 2017 .

DAI S N . Research on imperfect information game based on counterfactual regret minimization algorithm [D ] . Harbin :Harbin Institute of Technology , 2017 .

SKLANSKY D . The theory of poker [M ] . Two plus two publishing LLC , 1999 .

SKLANSKY D , MALMUTH M . Hold'em poker for advanced players [M ] . Two Plus Two Publishing LLC , 1999 .

JOHANSON M . Measuring the size of large no-limit poker games [J ] . arXiv preprint arXiv:1302.7008 , 2013 .

BORGHETTI B J . Opponent modeling in interesting adversarial environments [M ] . Minnesota : University of Minnesota , 2008 .

WRIGHT J R . Modeling human behavior in strategic settings [D ] . COLUMBIA:University of British Columbia , 2016 .

PLONSKY O , APEL R , ERT E , et al . Predicting human decisions with behavioral theories and machine learning [J ] . arXiv preprint arXiv:1904.06866 , 2019 .

高巍 , 罗俊仁 , 袁唯淋 , 等 . 面向对手建模的意图识别方法综述 [J ] . 网络与信息安全学报 , 2021 , 7 ( 4 ): 86 - 100 .

GAO W , LUO J R , YUAN W L , et al . A survey of intention recognition for opponent modeling [J ] . Chinese Journal of Network and Information Security , 2021 , 7 ( 4 ): 86 - 100 .

HOEHN B , SOUTHEY F , HOLTE R C , et al . Effective short-term opponent exploitation in simplified poker [C ] // AAAI . 2005 : 783 - 788 .

GANZFRIED S , SANDHOLM T . Game theory-based opponent modeling in large imperfect-information games [C ] // The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 . 2011 : 533 - 540 .

NORRIS K , WATSON I . A statistical exploitation module for Texas Hold'em:and it's benefits when used with an approximate Nash equilibrium strategy [C ] // 2013 IEEE Conference on Computational Inteligence in Games (CIG) . 2013 : 1 - 8 .

GANZFRIED S , SANDHOLM T . Safe opponent exploitation [J ] . ACM Transactions on Economics and Computation (TEAC) , 2015 , 3 ( 2 ): 1 - 28 .

SANDHOLM T . The state of solving large incomplete-information games,and application to poker [J ] . AI Magazine , 2010 , 31 ( 4 ): 13 - 32 .

SANDHOLM T . Solving imperfect-information games [J ] . Science , 2015 , 347 ( 6218 ): 122 - 123 .

SANDHOLM T , . Abstraction for solving large incomplete-information games [C ] // 29 AAAI Conference on Artificial Intelligence . 2015 .

GILPIN A , SANDHOLM T . Lossless abstraction of imperfect information games [J ] . Journal of the ACM (JACM) , 2007 , 54 ( 5 ).

GILPIN A , SANDHOLM T . Better automated abstraction techniques for imperfect information games,with application to Texas Hold’em poker [C ] // Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems . 2007 : 1 - 8 .

NOAM B , GANZFRIED S , SANDHOLM T . Hierarchical abstraction,distributed equilibrium computation,and post-processing,with application to a champion no-limit Texas Hold'em agent [C ] // Workshops at the 29 AAAI Conference on Artificial Intelligence . 2015 .

BROWN N , SANDHOLM T . Safe and nested subgame solving for imperfect-information games [C ] // Advances in Neural Information Processing Systems . 2017 : 689 - 699 .

NESTEROV . Excessive gap technique in nonsmooth convex minimization [J ] . SIAM Journal of Optimization , 2005 , 16 ( 1 ): 235 - 249 .

HART S A , MAS D , COLELL A . A simple adaptive procedure leading to correlated equilibrium [J ] . Econometrica , 2000 , 68 ( 5 ): 1127 - 1150 .

GREENWALD A , LI Z , MARKS C . Bounds for regret-matching algorithms [C ] // Proceedings of the 9 International Symposium on Artificial Intelligence and Mathematics . 2005 .

GIBSON R G , LANCTOT M , BURCH N , et al . Generalized sampling and variance in counterfactual regret minimization [C ] // Proceedings of the Twenty-Sixth Conference on Artificial Intelligence . 2012 : 3690 - 3706 .

LISY V , LANCTOT M , BOWLING M . Online monte carlo counterfactual regret minimization for search in imperfect information games [C ] // Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems . 2015 : 27 - 36 .

GANZFRIED S , SANDHOLM T . Endgame solving large imperfect-information games [C ] // Workshops at the Twenty-Ninth AAAAI Conference on Artificial Intelligence . 2015 .

BROWN N , SANDHOLM T . Solving imperfect-information games via discounted regret minimization [C ] // AAAI Conference on Artificial Intelligence (AAAI) . 2019 .

BROWN N , SANDHOLM T . Regret-based pruning in extensive form games [C ] // Advances in Neural Information Processing Systems . 2015 : 1972 - 1980 .

BROWN N , SANDHOLM T . Strategy-based warm starting for regret minimization in games [C ] // Proceedings of 30 Conference on Artificial Intelligence . 2015 : 432 - 438 .

LANCTOT M , WAUGH K , ZINKEVICH M , et al . Monte Carlo sampling for regret minimization in extensive games [C ] // Advances in Neural Information Processing Systems . 2009 : 1078 - 1086 .

LI H , HU K , GE Z , et al . Double neural counterfactual regret minimization [J ] . 2018 :

NOAM B , LERER A , GROSS S , et al . Deep counterfactual regret minimization [J ] . axXiv Preprint arXiv:1812.10607 , 2018 .

ERIC S . Single deep counterfactual regret minimization [J ] . arXiv Preprint arXiv:1901.07621 , 2018 .

FARINA G , KROER C , BROWN N , et al . Stable-predictive optimistic counterfactual regret minimization [C ] // International Conference on Machine Learning . 2019 : 1853 - 1862 .

陈学松 , 杨宜民 . 强化学习研究综述 [J ] . 计算机应用研究 , 2010 ( 8 ): 40 - 44 , 50 .

YANG X S , CHEN Y M . Reinforcement learning：survey of recent work [J ] . Application Research of Computers , 2010 ( 8 ): 40 - 44 , 50 .

HEINRICH J , SILVER D . Deep reinforcement learning from self-play in imperfect-information games [J ] . arXiv preprint arXiv:1603.01121 , 2016 .

LI K , XU H , ZHANG M , et al . OpenHoldem:an open toolkit for large-scale imperfect-information game research [J ] . arXiv preprint arXiv:2012.06168 , 2020 .

ZHA D , LAI K H , CAO Y , et al . RLCard:a toolkit for reinforcement learning in card games [J ] . arXiv preprint arXiv:1910.04376 , 2019 .

NIKOLAI Y , CAO L , RAFFEL C , et al . Poker-CNN:a pattern learning strategy for making draws and bets in poker games using convolutional networks [C ] // 30 AAAI Conference on Artificial Intelligence . 2016 .

KORB K B , NICHOLSON A , JITNAH N . Bayesian poker [J ] . arXiv preprint arXiv:1301.6711 , 2013 .

GANZFRIED S , SANDHOLM T . Endgame solving in large imperfect information games [C ] // Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems . 2015 : 37 - 45 .

SOUTHEY F , BOWLING M P , LARSON B , et al . Bayes' bluff:opponent modelling in poker [J ] . arXiv preprint arXiv:1207.1411 , 2012 .

LI H , WANG X , JIA F , et al . RLCFR:minimize counterfactual regret by deep reinforcement learning [J ] . arXiv preprint arXiv:2009.06373 , 2020 .

YANNAKAKIS G N , TOGELIUS J . Artificial intelligence and games [M ] . New York : Springer , 2018 .

Views

1211

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰