基于分布式内存计算的深度学习方法

doi:10.13229/j.cnki.jdxbgxb201503034

吉林大学学报(工学版) ›› 2015, Vol. 45 ›› Issue (3): 921-925.doi: 10.13229/j.cnki.jdxbgxb201503034

基于分布式内存计算的深度学习方法

李抵非¹, 田地¹, 胡雄伟²

1.吉林大学仪器科学与电气工程学院,长春 130021;
2.国家标准化管理委员会标准信息中心,北京100088

收稿日期:2014-10-09 出版日期:2015-05-01 发布日期:2015-05-01
通讯作者: 田地(1958-),男,教授,博士生导师.研究方向:分析仪器测控技术及软件.E-mail:tiandi@jlu.edu.cn E-mail:lidf12@mails.jlu.edu.cn
作者简介:李抵非(1986-),男,博士研究生.研究方向:人工智能技术.
基金资助:
科技部创新方法工作专项项目(2011IM010400)

A method of deep learning based on distributed memory computing

LI Di-fei¹, TIAN Di¹, HU Xiong-wei²

1.College of Instrumentation & Electrical Engineering, Jilin University, Changchun 130021, China;
2.Standardization Administration Information Center, Standardization Administration of the People's Republic of China, Beijing 100088, China

Received:2014-10-09 Online:2015-05-01 Published:2015-05-01

摘要/Abstract

摘要： 为了提升深度学习技术并行化学习效率,设计了一种面向计算机集群的分布式内存计算方法。构建分布式内存环境,建立数据分片处理和多任务调度机制,使模型参数和神经元节点的计算和存储并行运行于该环境中,避免了磁盘I/O对训练速率的影响;采用深度信念网络模型,以多个副本异步并行计算的方式进行模型训练,并使用dropout方法防止模型训练过拟合。对CIFAR-10图像数据集进行分类训练,试验结果表明,该方法可以明显提高深度神经网络的训练效率,并具备良好的可扩展性。

关键词: 人工智能, 并行深度学习, 分布式内存计算, 深信度网络

Abstract: To improve the efficiency of deep neural network distributed training, a new method is proposed, which makes neural network model running on distributed memory computing system. A framework of distributed memory is built, which contains functions of data partition and multi-task schedule. It can avoid the impact of I/O on the training rate and makes the training process run at memory-speed across cluster. Within the framework, multiple model replicas of deep believe network are trained in an asynchronous way. In addition, the dropout algorithm is employed to prevent over-fitting. The proposed method is evaluated using CIFAR-10 dataset. Experiment results show that the new method improves the efficiency of training deep neural network and enables scalability.

Key words: artificial intelligence, distributed deep learning, distributed memory computing, deep belief network

中图分类号:

TP183

李抵非, 田地, 胡雄伟. 基于分布式内存计算的深度学习方法[J]. 吉林大学学报(工学版), 2015, 45(3): 921-925.

LI Di-fei, TIAN Di, HU Xiong-wei. A method of deep learning based on distributed memory computing[J]. 吉林大学学报(工学版), 2015, 45(3): 921-925.

参考文献

[1] Ciresan D C, Meier U, Gambardella L M, et al. Deep big simple neural nets excel on handwritten digit recognition[J]. Neural Computation,2010(12):3207-3220.
[2] Coates A,Lee H L,Ng A Y.An analysis of single-layer networks in unsupervised feature learning[C]∥Proceeding of the 14th International Conference on Artificial Intelligence and Statistics,Fort Lauderdale,USA,2011:215-223.
[3] Hinton G E, Srivastava N, Krizhevsky A, et al. Improving neural networks by preventing co-adaptation of feature detectors[DB/OL].[2014-05-17].http://arxiv.org/abs/1207.0580.
[4] Raina R, Madhavan A,Ng A Y.Large-scale deep unsupervised learning using graphics processors[C]∥International Conference on Machine Learning,Montreal QC,Canada,2009:873-880.
[5] Le Q V,Monga R,Devin M,et al. Building high-level features using large scale unsupervised learning[C]∥International Conference on Acoustics, Speech and Signal,Vancouver,Canada,2013:8595-8598.
[6] Glorot X, Bordes A, Bengio Y. Domain adaptation for large-scale sentiment classification: a deep learning approach[C]∥Proceedings of the 28th International Conference on Machine Learning,Bellevue, WA,USA,2011:513-520.
[7] Bengio Y, Courville A C, Vincent P. Representation learning:a review and new perspectives[DB/OL].[2014-05-23].http://arxiv.org/abs/1206.5538.
[8] Ngiam J, Coates A, Lahiri A, et al. On optimization methods for deep learning[C]∥Proceedings of the 28th International Conference on Machine Learning,Bellevue, WA,USA,2011:265-272.
[9] Martens J. Deep learning via hessian-free optimization[C]∥Proceedings of the 27th International Conference on Machine Learning,Haifa,Israel,2010:735-742.
[10] Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(8):1798-1828.
[11] Hinton G, Deng L, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups[J]. Signal Processing Magazine,2012,29(6):82-97.
[12] Hinton G,Osindero S,Teh Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation,2006,18(7):1527-1554.
[13] Hinton G. A practical guide to training restricted Boltzmann machines[J]. Momentum,2010,9(1):926.
[14] Zaharia M,Chowdhury M,Das T,et al. Resilient distributed datasets:a fault-tolerant abstraction for in-memory cluster computing[DB/OL].[2014-01-19].http://www.eecs.berkeley.edu/Pubs/TechRpts/2011/EECS-2011-82.pdf.
[15] Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization[J]. Journal of Machine Learning Research,2011,12(7):2121-2159.

相关文章 15

[1]	董飒, 刘大有, 欧阳若川, 朱允刚, 李丽娜. 引入二阶马尔可夫假设的逻辑回归异质性网络分类方法[J]. 吉林大学学报(工学版), 2018, 48(5): 1571-1577.
[2]	顾海军, 田雅倩, 崔莹. 基于行为语言的智能交互代理[J]. 吉林大学学报(工学版), 2018, 48(5): 1578-1585.
[3]	王旭, 欧阳继红, 陈桂芬. 基于垂直维序列动态时间规整方法的图相似度度量[J]. 吉林大学学报(工学版), 2018, 48(4): 1199-1205.
[4]	张浩, 占萌苹, 郭刘香, 李誌, 刘元宁, 张春鹤, 常浩武, 王志强. 基于高通量数据的人体外源性植物miRNA跨界调控建模[J]. 吉林大学学报(工学版), 2018, 48(4): 1206-1213.
[5]	黄岚, 纪林影, 姚刚, 翟睿峰, 白天. 面向误诊提示的疾病-症状语义网构建[J]. 吉林大学学报(工学版), 2018, 48(3): 859-865.
[6]	李雄飞, 冯婷婷, 骆实, 张小利. 基于递归神经网络的自动作曲算法[J]. 吉林大学学报(工学版), 2018, 48(3): 866-873.
[7]	刘杰, 张平, 高万夫. 基于条件相关的特征选择方法[J]. 吉林大学学报(工学版), 2018, 48(3): 874-881.
[8]	王旭, 欧阳继红, 陈桂芬. 基于多重序列所有公共子序列的启发式算法度量多图的相似度[J]. 吉林大学学报(工学版), 2018, 48(2): 526-532.
[9]	杨欣, 夏斯军, 刘冬雪, 费树岷, 胡银记. 跟踪-学习-检测框架下改进加速梯度的目标跟踪[J]. 吉林大学学报(工学版), 2018, 48(2): 533-538.
[10]	刘雪娟, 袁家斌, 许娟, 段博佳. 量子k-means算法[J]. 吉林大学学报(工学版), 2018, 48(2): 539-544.
[11]	曲慧雁, 赵伟, 秦爱红. 基于优化算子的快速碰撞检测算法[J]. 吉林大学学报(工学版), 2017, 47(5): 1598-1603.
[12]	李嘉菲, 孙小玉. 基于谱分解的不确定数据聚类方法[J]. 吉林大学学报(工学版), 2017, 47(5): 1604-1611.
[13]	邵克勇, 陈丰, 王婷婷, 王季驰, 周立朋. 无平衡点分数阶混沌系统全状态自适应控制[J]. 吉林大学学报(工学版), 2017, 47(4): 1225-1230.
[14]	王生生, 王创峰, 谷方明. OPRA方向关系网络的时空推理[J]. 吉林大学学报(工学版), 2017, 47(4): 1238-1243.
[15]	马淼, 李贻斌. 基于多级图像序列和卷积神经网络的人体行为识别[J]. 吉林大学学报(工学版), 2017, 47(4): 1244-1252.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于分布式内存计算的深度学习方法

A method of deep learning based on distributed memory computing

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

Metrics

本文评价

推荐阅读 10