关键词: 分布式计算, MapReduce 模型, 特征投影, 多项式拟合, 频繁模式, 并行挖掘

Abstract: Considering the characteristics and complexity of multidimensional data streams, in order to fully utilize parallel computing resources and ensure the scalability of the algorithm, a parallel mining algorithm for frequent patterns of multidimensional data streams in Hadoop environment is proposed. Design a Hadoop data stream processing platform based on HDFS ( Hadoop Distributed File System) and MapReduce, propose an HpFitStream clustering algorithm based on feature projection and fitting, using the polynomial fitting algorithm to handle abnormal data streams, and reducing the dimensionality of the processed data streams through feature projection to reduce computational costs. Implement frequent pattern parallel mining of multidimensional data streams in Hadoop environment using PFPonCanTree algorithm. The experimental results show that the proposed method can effectively reduce computational complexity while improving the scalability and load balancing ability of the algorithm.

Key words: Hadoop, MapReduce model, feature projection, polynomial fitting, frequent mode, parallel mining