Abstract:

We proposed a scalable and dynamic MapReduce computation model which supports the online processing of largescale dynamic/static data against the characteristics of uneven stream data size and dynamic flowing  and breaking out suddenly. On this basis, we proposed an online MapReduce data transmission mechanism and implemented its prototype program based on the push mode of Event and the use of Netty asynchronous communication technology. This paper focuses on solving fast online transfer of the largescale distributed computing program and data dynamic distribution to provide support for dynamic MapReduce model. The experimental results show that the method can greatly improve the transmission efficiency of data between jobs compared with the traditional socket pipeline method in Hadoop system and improve realtime data stream handling significantly.

Key words: big data, stream data processing, MapReduce model, data transmission mechanism