吉林大学学报(工学版) ›› 2016, Vol. 46 ›› Issue (4): 1222-1231.doi: 10.13229/j.cnki.jdxbgxb201604031

• Orginal Article • Previous Articles     Next Articles

Flexible Online MapReduce model and topology protocols supporting large-scale stream data processing

WEI Xiao-hui1, 2, LI Xiang1, LI Hong-liang1, 2, LI Cong1, ZHUANG Yuan1, YU Hong-mei1   

  1. 1.College of Computer Science and Technology,Jilin University, Changchun 130012,China;
    2.Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012,China
  • Received:2015-05-15 Online:2016-07-20 Published:2016-07-20

RICH HTML

0   

PDF (PC)

109

Abstract: To meet the rapid growing requirements of large-scale data online processing, this paper proposes a Flexible Online MapReduce model and related dynamic topology protocols for streaming data processing. This model is compatible with existing MapReduce application, adopts in-memory computing, and possesses a dynamic job topology. It can support flexible adjustments of large-scale stream data in runtime to meet the requirements of real-time processing, dynamic flow and burst of data. Based on Flexible Online MapReduce model, the system architecture is designed to facilitate the model and a series of protocols are introduced, including Online Topology Initialization Protocol (OTIP), and Online Dynamic Adjusting Protocol (ODAP) for stream data job. To further integrate the system resources, the concept of operation sharing is introduced and Job Sharing Protocol (JSP) is designed. Protocol analyses illustrate that the communicating complexity of the protocol is O(n), and the system is able to adapt to the burst of stream data.

Key words: computer system organization, stream processing, memory computing, MapReduce, topology structure, job sharing

CLC Number: 

  • TP391
[1] Dean J,Ghemawat S. MapReduce: simplified data processing on large clusters[J].Commun ACM, 2008,51(1):107-113.
[2] Apache Hadoop[EB/OL].[2012-10-10].http:∥hadoop.apache.org/
[3] Isard B M, Budiy M,Yu Y,et al. Dryad: Distributed data-parallel programs from sequential building blocks[J]. Proceedings of ACM SIGOPS Operating Systems Review, 2007,41(3):59-72.
[4] Lam W, Liu L, Prasad S T S, et al. Muppet: MapReduce-style processing of fast data[J]. Proceedings of the VLDB Endowment, 2012, 5(12): 1814-1825.
[5] Brito A, Martin A, Knauth T, et al. Scalable and low-latency data processing with stream MapReduce[C]∥IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), 2011: 48-58.
[6] Gannon D,Deelman E,Shields M, et al.Workflows for e-Science.Introduction[M].Berlin:Springer,2007:1-9.
[7] Nykiel T,Potamias M, Mishra C,et al.MRShare: sharing across multiple queries in MapReduce[J].Proceedings of the VLDB Endowment,2010,3(1-2):494-505.
[8] Nykiel T, Potamias M, Mishra C, et al. Sharing across Multiple MapReduce Jobs[J]. Acm Transactions on Database Systems, 2014, 39(2):1-46.
[9] Luckham D C. The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems[M]. Boston, MA, USA: Addison-Wesley Longman Publishing Co, Inc, 2001.
[10] Marz N. Storm-distributed and fault-tolerant realtime computation[EB/OL].http:∥storm-project. net/,2013-02-01.
[11] Neumeyer L, Robbins B, Nair A, et al. S4: distributed stream computing platform[C]∥Proceedings of IEEE International Conference on Data Mining Workshops, 2010:170-177.
[12] Condie T, Conway N, Alvaro P, et al. MapReduce online[J].Proceedings of the 7th USENIX Symposium on Networked Systems Design and Implementation, 2010,10(4):313-328.
[13] Backman N, Pattabiraman K, Fonseca R, et al. C-MR: continuously executing MapReduce workflows on multi-core processors[C]∥International Workshop on Mapreduce and ITS Applications, Mapreduce,2012:1-8.
[1] ZHAO Bo, QIN Gui-He, ZHAO Yong-Zhe, YANG Wen-Di. Public key cryptosystem based on semi-trapdoor one-way function [J]. 吉林大学学报(工学版), 2018, 48(1): 259-267.
[2] YU Bin-bin, WU Xin-yu, CHU Jian-feng, HU Liang. Signature protocol for wireless sensor network based on group key agreement [J]. 吉林大学学报(工学版), 2017, 47(3): 924-929.
[3] CHE Xiang-jiu, LIANG Sen. Improved algorithm of SPIHT based on Max-Heap tree [J]. 吉林大学学报(工学版), 2016, 46(3): 865-869.
[4] LI Di-fei, TIAN Di, HU Xiong-wei. A method of deep learning based on distributed memory computing [J]. 吉林大学学报(工学版), 2015, 45(3): 921-925.
[5] FU Shuai, MA Jian-feng, LI Hong-tao, WANG Chang-guang. Improved data aggregation algorithm based on clustered wireless sensor network [J]. 吉林大学学报(工学版), 2014, 44(4): 1118-1125.
[6] HU Liang, CHI Ling, YUAN Wei, CHU Jian-feng, XU Xiao-bo. Improvements against fault induction attack for RC4 algorithm [J]. , 2012, 42(05): 1231-1236.
[7] LIU San-min, SUN Zhi-xin. P2P traffic identification based on support vector data description [J]. , 2012, 42(04): 947-951.
[8] ZHANG Rui-Hua, CHENG He-You, JIA Zhi-Beng. Energyefficient clustering algorithm for wireless sensor networks [J]. 吉林大学学报(工学版), 2010, 40(06): 1663-1667.
[9] LIU Yan-Hang, SUN Xin, WANG Jian, LI Wei-Ping, ZHU Jian-Qi. Emailworm propagation with user behavior and network topology [J]. 吉林大学学报(工学版), 2010, 40(06): 1655-1662.
[10] WEI Da, Gu-Xiang-Peng, WANG Jian, Liu-Yan-Hang. New access model and implementation of trusted network based on trusted certificate [J]. 吉林大学学报(工学版), 2010, 40(02): 496-0500.
[11] LIU Yan-Heng, SUN Lei, TIAN Da-Xin, WU Jing. Method of anomaly detection based on fusion principal components match [J]. 吉林大学学报(工学版), 2009, 39(05): 1314-1320.
[12] WANG Xiao-yan,LIU Shu-fen,YU Hai . Interface automata based approach to Web service composition [J]. 吉林大学学报(工学版), 2009, 39(03): 743-0748.
[13] MEI Fang，LIU Yan-heng，ZHANG Xu-li，Gu Tian-ye，WANG Wang . Dynamic conflict resolution mechanism for resource
management policy in mobile network
 [J]. 吉林大学学报(工学版), 2009, 39(02): 430-0435.
[14] SUN Zhi-xin, YANG Xi, GONG Jing . New P2P architecture description language [J]. 吉林大学学报(工学版), 2008, 38(05): 1131-1135.
[15] Yang Chao，Cao Chun-jie，Wang Wei，Ma Jian-feng . New authentication protocol of roaming for wireless mesh network [J]. 吉林大学学报(工学版), 2008, 38(02): 423-0428.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] LIU Song-shan, WANG Qing-nian, WANG Wei-hua, LIN Xin. Influence of inertial mass on damping and amplitude-frequency characteristic of regenerative suspension[J]. 吉林大学学报(工学版), 2013, 43(03): 557 -563 .
[2] CHU Liang, WANG Yan-bo, QI Fu-wei, ZHANG Yong-sheng. Control method of inlet valves for brake pressure fine regulation[J]. 吉林大学学报(工学版), 2013, 43(03): 564 -570 .
[3] LI Jing, WANG Zi-han, YU Chun-xian, HAN Zuo-yue, SUN Bo-hua. Design of control system to follow vehicle state with HIL test beach[J]. 吉林大学学报(工学版), 2013, 43(03): 577 -583 .
[4] HU Xing-jun, LI Teng-fei, WANG Jing-yu, YANG Bo, GUO Peng, LIAO Lei. Numerical simulation of the influence of rear-end panels on the wake flow field of a heavy-duty truck[J]. 吉林大学学报(工学版), 2013, 43(03): 595 -601 .
[5] WANG Tong-jian, CHEN Jin-shi, ZHAO Feng, ZHAO Qing-bo, LIU Xin-hui, YUAN Hua-shan. Mechanical-hydraulic co-simulation and experiment of full hydraulic steering systems[J]. 吉林大学学报(工学版), 2013, 43(03): 607 -612 .
[6] ZHANG Chun-qin, JIANG Gui-yan, WU Zheng-yan. Factors influencing motor vehicle travel departure time choice behavior[J]. 吉林大学学报(工学版), 2013, 43(03): 626 -632 .
[7] MA Wan-jing, XIE Han-zhou. Integrated control of main-signal and pre-signal on approach of intersection with double stop line[J]. 吉林大学学报(工学版), 2013, 43(03): 633 -639 .
[8] YU De-xin, TONG Qian, YANG Zhao-sheng, GAO Peng. Forecast model of emergency traffic evacuation time under major disaster[J]. 吉林大学学报(工学版), 2013, 43(03): 654 -658 .
[9] XIAO Yun, LEI Jun-qing, ZHANG Kun, LI Zhong-san. Fatigue stiffness degradation of prestressed concrete beam under multilevel amplitude cycle loading[J]. 吉林大学学报(工学版), 2013, 43(03): 665 -670 .
[10] XIAO Rui, DENG Zong-cai, LAN Ming-zhang, SHEN Chen-liang. Experiment research on proportions of reactive powder concrete without silica fume[J]. 吉林大学学报(工学版), 2013, 43(03): 671 -676 .
Copyright © Journal of Jilin University(Engineering and Technology Edition), All Rights Reserved.
Tel: +86-431-85095297　Fax: +86-431-85094074　E-mail: xbgxb@jlu.edu.cn
Powered by Beijing Magtech Co. Ltd