吉林大学学报(工学版) ›› 2015, Vol. 45 ›› Issue (2): 596-599.doi: 10.13229/j.cnki.jdxbgxb201502038

• Orignal Article • Previous Articles     Next Articles

Standard literature language model based on deep learning

LI Di-fei1, TIAN Di1, HU Xiong-wei2   

  1. 1.College of Instrumentation &
    Electrical Engineering, Jilin University, Changchun 130021, China;
    2.Standardization Administration Information Center, Standardization Administration of the People's Republic of China, Beijing 100088, China
  • Received:2013-11-07 Online:2015-04-01 Published:2015-04-01

Abstract: To solve the problem of natural language processing for Chinese standard literature, the deep learning technology is employed to build a statistical language model. The Hierarchical Log-Bilinear language model is improved and the unsupervised learning and supervised learning are integrated. In order to accomplish the machine learning, the stacked restricted Boltzman machines are taken to train words' distributed representations, which are taken as the input to a supervised feedforward neural network. The proposed is evaluated using more than one million standard literature bibliographic data. Experiment results show that this model can effectively improve the model's ability to learn the probability of words' distribution.

Key words: artificial intelligence, natural language processing, statistical language model, deep neural networks, restricted boltzman machines, distributed representations

CLC Number: 

  • TP183
[1] Blei David M, Ng Andrew, Jordan Michael. Latent dirichlet allocation[J]. JMLR, 2003,3:993-1022.
[2] Blei David M, Griffths Thomas L, Jordan Michael I. The nested chinese restaurant process and bayesian nonparametric inference of topic hierarchies[J]. ACM, 2010,57(2):7-38.
[3] Mimno D, McCallum A. Topic models conditioned on arbitrary features with dirichlet-multinomial regression[J].UAI, 2008:411-418.
[4] Salakhutdinov Ruslan, Hinton Georey. Replicated softmax: an undirected topic model[J].In Advances in Neural Information Processing Systems, 2009, 22:1607-1614.
[5] Bengio Yoshua, Ducharme Réjean, Vincent Pascal, et al. A neural probabilistic language model[J].Journal of Machine Learning Research, 2003 (3): 1137-1155.
[6] Collobert Ronan, Weston Jason. Natural language processing (almost) from scratch[J].Journal of Machine Learning Research, 2000 (1): 1-48.
[7] Mnih Andriy, Hinton Geoffrey.Three new graphical models for statistical language modelling[C]∥International Conference on Machine Learning,Oregan,USA,2007:641-648.
[8] Mnih Andriy, Hinton Geoffrey.A scalable hierarchical distributed language model[C]∥Conference on Neural Information Processing Systems,Canada,2008:1081-1088.
[9] Srivastava N, Salakhutdinov R R, Hinton G E. Modeling documents with a deep boltzmann machine in uncertainty in artificial intelligence[C]∥The Conference on Uncertainty in Artificial Intelligence,Bellevue,Washiugton,USA,2013:1309-1318.
[10] Mikolov T, Karafiát M, Burget L, et al. Recurrent neural network based language model[C]∥Interspeech,Makuhari,Japan,2010:1045-1048.
[11] Huang Eric H, Socher Richard, Manning Christopher D. Improving word representations via global context and multiple word prototypes[C]∥Association for Computational Linguistics,Stroudsburg,PA,USA,2012:873-882.
[12] Xue N. Chinese word segmentation as character tagging[J]. Computational Linguistics and Chinese Language Processing, 2003, 8(1): 29-48.
[13] Tang B, Wang X, Wang X. Chinese word segmentation based on large margin nethods[J]. Int J of Asian Lang Proc, 2009, 19(2): 55-68.
[14] Zhao H, Kit C. Integrating unsupervised and supervised word segmentation: the role of goodness measures[J]. Information Sciences, 2011, 181(1): 163-183.
[15] Hinton G. A practical guide to training restricted Boltzmann machines[J]. Momentum, 2010, 9(1): 926.
[16] Bengio Y. Learning deep architectures for AI[J]. Foundations and Trends in Machine Learning, 2009, 2(1): 1-127.
[17] Hinton G E,Salakhutdinov R R. Reducing the dimensionality of data with neural networks[J].Science, 2006, 313(5786): 504-507.
[18] Bengio Y, Lamblin P, Popovici D, et al. Greedy layer-wise training of deep networks[J]. Advances in Neural Information Processing Systems, 2007, 19: 153.
[19] Graves A, Mohamed A, Hinton G. Speech recognition with deep recurrent neural networks[C]∥Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2013: 6645-6649.
[20] Tieleman T. Training restricted Boltzmann machines using approximations to the likelihood gradient[C]∥Proceedings of the 25th International Conference on Machine Learning,ACM, 2008: 1064-1071.
[21] Trurian J, Ratinov L,Bengio Y.Word representations:a simple and general method for semi-supervised learning[C]∥Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics,Stroudsburg,PA,USA,2010:384-394.
[1] DONG Sa, LIU Da-you, OUYANG Ruo-chuan, ZHU Yun-gang, LI Li-na. Logistic regression classification in networked data with heterophily based on second-order Markov assumption [J]. Journal of Jilin University(Engineering and Technology Edition), 2018, 48(5): 1571-1577.
[2] GU Hai-jun, TIAN Ya-qian, CUI Ying. Intelligent interactive agent for home service [J]. Journal of Jilin University(Engineering and Technology Edition), 2018, 48(5): 1578-1585.
[3] WANG Xu, OUYANG Ji-hong, CHEN Gui-fen. Measurement of graph similarity based on vertical dimension sequence dynamic time warping method [J]. 吉林大学学报(工学版), 2018, 48(4): 1199-1205.
[4] ZHANG Hao, ZHAN Meng-ping, GUO Liu-xiang, LI Zhi, LIU Yuan-ning, ZHANG Chun-he, CHANG Hao-wu, WANG Zhi-qiang. Human exogenous plant miRNA cross-kingdom regulatory modeling based on high-throughout data [J]. 吉林大学学报(工学版), 2018, 48(4): 1206-1213.
[5] HUANG Lan, JI Lin-ying, YAO Gang, ZHAI Rui-feng, BAI Tian. Construction of disease-symptom semantic net for misdiagnosis prompt [J]. 吉林大学学报(工学版), 2018, 48(3): 859-865.
[6] LI Xiong-fei, FENG Ting-ting, LUO Shi, ZHANG Xiao-li. Automatic music composition algorithm based on recurrent neural network [J]. 吉林大学学报(工学版), 2018, 48(3): 866-873.
[7] LIU Jie, ZHANG Ping, GAO Wan-fu. Feature selection method based on conditional relevance [J]. 吉林大学学报(工学版), 2018, 48(3): 874-881.
[8] WANG Xu, OUYANG Ji-hong, CHEN Gui-fen. Heuristic algorithm of all common subsequences of multiple sequences for measuring multiple graphs similarity [J]. 吉林大学学报(工学版), 2018, 48(2): 526-532.
[9] YANG Xin, XIA Si-jun, LIU Dong-xue, FEI Shu-min, HU Yin-ji. Target tracking based on improved accelerated gradient under tracking-learning-detection framework [J]. 吉林大学学报(工学版), 2018, 48(2): 533-538.
[10] LIU Xue-juan, YUAN Jia-bin, XU Juan, DUAN Bo-jia. Quantum k-means algorithm [J]. 吉林大学学报(工学版), 2018, 48(2): 539-544.
[11] QU Hui-yan, ZHAO Wei, QIN Ai-hong. A fast collision detection algorithm based on optimization operator [J]. 吉林大学学报(工学版), 2017, 47(5): 1598-1603.
[12] LI Jia-fei, SUN Xiao-yu. Clustering method for uncertain data based on spectral decomposition [J]. 吉林大学学报(工学版), 2017, 47(5): 1604-1611.
[13] SHAO Ke-yong, CHEN Feng, WANG Ting-ting, WANG Ji-chi, ZHOU Li-peng. Full state based adaptive control of fractional order chaotic system without equilibrium point [J]. 吉林大学学报(工学版), 2017, 47(4): 1225-1230.
[14] WANG Sheng-sheng, WANG Chuang-feng, GU Fang-ming. Spatio-temporal reasoning for OPRA direction relation network [J]. 吉林大学学报(工学版), 2017, 47(4): 1238-1243.
[15] MA Miao, LI Yi-bin. Multi-level image sequences and convolutional neural networks based human action recognition method [J]. 吉林大学学报(工学版), 2017, 47(4): 1244-1252.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] LIU Song-shan, WANG Qing-nian, WANG Wei-hua, LIN Xin. Influence of inertial mass on damping and amplitude-frequency characteristic of regenerative suspension[J]. 吉林大学学报(工学版), 2013, 43(03): 557 -563 .
[2] CHU Liang, WANG Yan-bo, QI Fu-wei, ZHANG Yong-sheng. Control method of inlet valves for brake pressure fine regulation[J]. 吉林大学学报(工学版), 2013, 43(03): 564 -570 .
[3] LI Jing, WANG Zi-han, YU Chun-xian, HAN Zuo-yue, SUN Bo-hua. Design of control system to follow vehicle state with HIL test beach[J]. 吉林大学学报(工学版), 2013, 43(03): 577 -583 .
[4] HU Xing-jun, LI Teng-fei, WANG Jing-yu, YANG Bo, GUO Peng, LIAO Lei. Numerical simulation of the influence of rear-end panels on the wake flow field of a heavy-duty truck[J]. 吉林大学学报(工学版), 2013, 43(03): 595 -601 .
[5] WANG Tong-jian, CHEN Jin-shi, ZHAO Feng, ZHAO Qing-bo, LIU Xin-hui, YUAN Hua-shan. Mechanical-hydraulic co-simulation and experiment of full hydraulic steering systems[J]. 吉林大学学报(工学版), 2013, 43(03): 607 -612 .
[6] ZHANG Chun-qin, JIANG Gui-yan, WU Zheng-yan. Factors influencing motor vehicle travel departure time choice behavior[J]. 吉林大学学报(工学版), 2013, 43(03): 626 -632 .
[7] MA Wan-jing, XIE Han-zhou. Integrated control of main-signal and pre-signal on approach of intersection with double stop line[J]. 吉林大学学报(工学版), 2013, 43(03): 633 -639 .
[8] YU De-xin, TONG Qian, YANG Zhao-sheng, GAO Peng. Forecast model of emergency traffic evacuation time under major disaster[J]. 吉林大学学报(工学版), 2013, 43(03): 654 -658 .
[9] XIAO Yun, LEI Jun-qing, ZHANG Kun, LI Zhong-san. Fatigue stiffness degradation of prestressed concrete beam under multilevel amplitude cycle loading[J]. 吉林大学学报(工学版), 2013, 43(03): 665 -670 .
[10] XIAO Rui, DENG Zong-cai, LAN Ming-zhang, SHEN Chen-liang. Experiment research on proportions of reactive powder concrete without silica fume[J]. 吉林大学学报(工学版), 2013, 43(03): 671 -676 .