Abstract:

To meet the requirements for online processing massive stream data, the authors proposed a novel system model, Flexible workflow, which is different from the traditional Map/Reduce stream data processing. This model conducts the online Map/Reduce parallelization of the process unit of workflow and executes a system of SPATE. A set of topology management protocol was designed for dynamic online Map/Reduce stream data processing model. The protocol includes a group of communication rules about setting up, managing and maintaining jobs. The experimental results validate the topology management protocol is effective, and can manage the Flexible workflow processing model availably.

Key words: steam processing, Flexible workflow model, Map/Reduce, topology management