吉林大学学报(工学版) ›› 2013, Vol. 43 ›› Issue (01): 135-140.

• 论文 • 上一篇    下一篇

基于多Agent并行采样和学习经验复用的E3算法

刘全1,2, 杨旭东1, 荆玲3, 肖飞1   

  1. 1. 苏州大学 计算机科学与技术学院, 江苏 苏州 215006;
    2. 吉林大学 符号计算与知识工程教育部重点实验室, 长春 130012;
    3. 南京大学 计算机科学与技术系, 南京 210093
  • 收稿日期:2012-03-11 出版日期:2013-01-01 发布日期:2013-01-01
  • 通讯作者: 杨旭东(1987-),男,硕士研究生.研究方向:机器学习,数据挖掘.E-mail:yangxudongsuda@gmail.com E-mail:yangxudongsuda@gmail.com
  • 作者简介:刘全(1969-),男,教授,博士生导师.研究方向:机器学习,自动推理.E-mail:quanliu@suda.edu.cn
  • 基金资助:

    国家自然科学基金项目(61070223,61103045,60970015,61170020,61272005);江苏省自然科学基金项目(BK2009116,BK2012616);江苏省高校自然科学研究项目(09KJA520002,09KJB520012);吉林大学符号计算与知识工程教育部重点实验室项目(93K172012K04).

Improved E3 algorithm based on multi-agent parallel sampling and learning experience reuse

LIU Quan1,2, YANG Xu-dong1, JING Ling3, XIAO Fei1   

  1. 1. School of Computer Science and Technology, Soochow University, Suzhou, 215006, China;
    2. Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China;
    3. Department of Computer Science and Technology, Nanjing University, Nanjing 210093, China
  • Received:2012-03-11 Online:2013-01-01 Published:2013-01-01

PDF (PC)

423

摘要： 针对E3算法所需的收敛时间界限太大,在实际问题中难以有效应用的问题,提出了一种基于多Agent并行采样和学习经验复用的改进算法。该算法在探索阶段,通过多Agent并行采样,快速收集模型信息,加速了模型构建过程;在利用阶段,通过保留最优值函数的方式复用算法的学习经验,提高了算法迭代计算值函数的效率。仿真实验结果表明,所提方法与原始的E3算法相比,在收敛速度和精度方面都具有很大的提高,与其他两种并行强化学习方法相比也具有很大的性能优势。

关键词: 人工智能, 强化学习, E3算法, 多Agent, 并行采样, 学习经验复用

Abstract: Existing E3 algorithm has the drawback of long convergence time, which can not be efficiently applied in practice. To overcome this problem, an improved E3 algorithm based on parallel sampling with multiple agents and learning experience reuse is proposed. In order to build an approximate model soon, in the exploration phase, the multiple agents explore the environment in parallel and collect the information of the environmental model. In the exploitation phase, the optimal value function is retained and reused to speed up the convergence. Experiment results show that the improved algorithm consumes much less to converge to an almost optimal policy and has great advantage over other two parallel reinforcement learning algorithms.

Key words: artificial intelligence, reinforcement learning, E3 algorithm, multi-agent, parallel sampling, learning experience reuse

中图分类号: 

  • TP181
[1] 徐心和,王骄. 中国象棋计算机博弈关键技术分析[J]. 小型微型计算机系统,2006,27(6): 961-969. Xu Xin-he, Wang Jiao. Key technologies analysis of Chinese Chess Computer Game[J]. Mini-Micro Systems, 2006, 27(6): 961-969.

[2] Kretchmar R M. Parallel reinforcement learning//Proceedings of the 6th World Conference on Systemics, Cybernetics, and Informatics. Orlando, Florida, USA, 2002: 114-118.

[3] Kretchmar R M. Reinforcement learning algorithms for homogenous multi-agent systems//Workshop on Agent and Swarm Programming, Cleveland, OH, USA, 2003.

[4] Printista A M, Errecalde M L, Montoya C I. A parallel implementation of Q-learning based on communication with cache[J]. Journal of Computer Science & Technology, 2002, 6: 268-278.

[5] Grounds M, Kudenko D. Parallel reinforcement learning with linear function approximation//Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), Honolulu, Hawai'i, USA, 2007.

[6] Kearns M J, Singh S P. Near-optimal reinforcement learning in polynomial time[J].Machine Learning, 2002, 49(2-3): 209-232.

[7] Brafman R I, Tennenholtz M. R-MAX-a general polynomial time algorithm for near-optimal reinforcement learning[J]. Journal of Machine Learning Research, 2002, 3:213-231.

[8] Strehl A L, Littman M L. A theoretical analysis of model-based interval estimation//Proceedings of the 22th International Conference on Machine Learning, Bonn, Germany, 2005: 857-864.

[9] Szita I, Lorincz A. The many faces of optimism: a unifying approach//Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, 2008: 1048-1055.

[10] Kakade S M. On the sample complexity of reinforcement learning.Gatsby Computational Neuroscience Unit, University College London, 2003.

[11] Szita I, Szepesvari C. Model-based reinforcement learning with nearly tight exploration complexity bounds//Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, 2010: 1031-1038.

[12] Domingo C.Faster near-optimal reinforcement learning:adding adaptiveness to the E3 algorithm//Proceedings of the 10th International Conference on Algorithmic Learning Theory,Volume 1720 of Lecture Notes in Computer Science,Springer.1999:241-251.

[13] Szepesvari C. Algorithms for Reinforcement Learning[M].Synthesis Lectures on Artifical Intelligence and Machine Learning. Morgan & Claypool Publishers, 2010.

[14] Wingate D, Seppi K D. P3VI: A partitioned, prioritized, parallel value iterator//Proceedings of the 21st International Conference on Machine Learning, Banff, Alberta, Canada, 2004: 109-116.
[1] 董飒, 刘大有, 欧阳若川, 朱允刚, 李丽娜. 引入二阶马尔可夫假设的逻辑回归异质性网络分类方法[J]. 吉林大学学报(工学版), 2018, 48(5): 1571-1577.
[2] 顾海军, 田雅倩, 崔莹. 基于行为语言的智能交互代理[J]. 吉林大学学报(工学版), 2018, 48(5): 1578-1585.
[3] 王旭, 欧阳继红, 陈桂芬. 基于垂直维序列动态时间规整方法的图相似度度量[J]. 吉林大学学报(工学版), 2018, 48(4): 1199-1205.
[4] 张浩, 占萌苹, 郭刘香, 李誌, 刘元宁, 张春鹤, 常浩武, 王志强. 基于高通量数据的人体外源性植物miRNA跨界调控建模[J]. 吉林大学学报(工学版), 2018, 48(4): 1206-1213.
[5] 黄岚, 纪林影, 姚刚, 翟睿峰, 白天. 面向误诊提示的疾病-症状语义网构建[J]. 吉林大学学报(工学版), 2018, 48(3): 859-865.
[6] 李雄飞, 冯婷婷, 骆实, 张小利. 基于递归神经网络的自动作曲算法[J]. 吉林大学学报(工学版), 2018, 48(3): 866-873.
[7] 刘杰, 张平, 高万夫. 基于条件相关的特征选择方法[J]. 吉林大学学报(工学版), 2018, 48(3): 874-881.
[8] 王旭, 欧阳继红, 陈桂芬. 基于多重序列所有公共子序列的启发式算法度量多图的相似度[J]. 吉林大学学报(工学版), 2018, 48(2): 526-532.
[9] 杨欣, 夏斯军, 刘冬雪, 费树岷, 胡银记. 跟踪-学习-检测框架下改进加速梯度的目标跟踪[J]. 吉林大学学报(工学版), 2018, 48(2): 533-538.
[10] 刘雪娟, 袁家斌, 许娟, 段博佳. 量子k-means算法[J]. 吉林大学学报(工学版), 2018, 48(2): 539-544.
[11] 曲慧雁, 赵伟, 秦爱红. 基于优化算子的快速碰撞检测算法[J]. 吉林大学学报(工学版), 2017, 47(5): 1598-1603.
[12] 李嘉菲, 孙小玉. 基于谱分解的不确定数据聚类方法[J]. 吉林大学学报(工学版), 2017, 47(5): 1604-1611.
[13] 邵克勇, 陈丰, 王婷婷, 王季驰, 周立朋. 无平衡点分数阶混沌系统全状态自适应控制[J]. 吉林大学学报(工学版), 2017, 47(4): 1225-1230.
[14] 王生生, 王创峰, 谷方明. OPRA方向关系网络的时空推理[J]. 吉林大学学报(工学版), 2017, 47(4): 1238-1243.
[15] 马淼, 李贻斌. 基于多级图像序列和卷积神经网络的人体行为识别[J]. 吉林大学学报(工学版), 2017, 47(4): 1244-1252.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
版权所有 © 《吉林大学学报(工学版)》编辑部
地址：长春市人民大街5988号　电话：+86-431-85095297　传真：+86-431-85094074　E-mail: xbgxb@jlu.edu.cn
本系统由北京玛格泰克科技发展有限公司设计开发