Abstract:

We proposed an operation sharing optimization algorithm based on the location aware according to the data flow in large scale, and the basic data processing operations had overlapping features. We assumed that the operations had assigned the resources of compute nodes in advance, in guaranteed the stream processing realtime and dynamic state, to take advantage of the similarity between topologies and operational functions of multiple stream data operations. We established an operation sharing revenue optimized model based on the DAG matching method on the constraint of output bandwidth. This method tackled the insufficiency consideration of traditional operation for operation sharing method to the cluster of output bandwidth. Simulation experiment results show that this method has fuller consideration in the calculation of the node constraints and improves the benefits of sharing, thereby saving the computing resources of the system effectively.

Key words: stream data, distributed stream processing, operation sharing, sharing optimization