动态在线Map/Reduce流数据处理模型及作业拓扑管理协议

吉林大学学报(理学版)

动态在线Map/Reduce流数据处理模型及作业拓扑管理协议

魏晓辉, 李翔, 李洪亮, 李聪, 庄园

吉林大学计算机科学与技术学院, 长春 130012

收稿日期:2014-08-27 出版日期:2015-09-26 发布日期:2015-09-29
通讯作者: 李洪亮 E-mail:lihongliang@jlu.edu.cn

Dynamic Online Map/Reduce Stream Processing Modeland Topology Management Protocol

WEI Xiaohui, LI Xiang, LI Hongliang, LI Cong, ZHUANG Yuan

College of Computer Science and Technology, Jilin University, Changchun 130012, China

Received:2014-08-27 Online:2015-09-26 Published:2015-09-29
Contact: LI Hongliang E-mail:lihongliang@jlu.edu.cn

摘要/Abstract

摘要：

针对海量流数据的在线处理需求, 提出一种不同于传统Map/Reduce流数据处理的系统模型Flexible workflow. 该模型对workflow处理单元进行在线Map/Reduce并行化, 实现了SPATE系统；同时为该系统定义一组关于作业的建立、管理和维护的通信规程, 即拓扑管理协议. SPATE系统解决了在线Map/Reduce流数据处理过程中要求实时性及可扩展性的问题. 实验验证了拓扑管理协议的有效性, 拓扑管理协议能有效管理Flexible workflow流数据处理模型.

关键词: 流数据处理, Flexible workflow模型, Map/Reduce, 拓扑管理

Abstract:

To meet the requirements for online processing massive stream data, the authors proposed a novel system model, Flexible workflow, which is different from the traditional Map/Reduce stream data processing. This model conducts the online Map/Reduce parallelization of the process unit of workflow and executes a system of SPATE. A set of topology management protocol was designed for dynamic online Map/Reduce stream data processing model. The protocol includes a group of communication rules about setting up, managing and maintaining jobs. The experimental results validate the topology management protocol is effective, and can manage the Flexible workflow processing model availably.

Key words: steam processing, Flexible workflow model, Map/Reduce, topology management

中图分类号:

TP311.11

魏晓辉, 李翔, 李洪亮, 李聪, 庄园. 动态在线Map/Reduce流数据处理模型及作业拓扑管理协议[J]. 吉林大学学报(理学版), 2015, 53(05): 950-955.

WEI Xiaohui, LI Xiang, LI Hongliang, LI Cong, ZHUANG Yuan. Dynamic Online Map/Reduce Stream Processing Modeland Topology Management Protocol[J]. Journal of Jilin University Science Edition, 2015, 53(05): 950-955.