Abstract:

In order to process huge amount of data generated in the social network with efficiency and scalability, we designed the distributed TF-IDF (Term Frequency-Inverse Document Frequency) algorithm suitable for MapReduce, and implemented this algorithm on Hadoop. This algorithm extracts key words in user's weibo, in this way user's interest could be found, and the corresponding service could be recommended to the user. In order to verify the validity and scalability of the distributed TFIDF algorithm, the results of the distributed TF-IDF algorithm and TextRank algorithm was compared. The experimental results show that key words extracted by the distributed TF-IDF algorithm could represent characteristics of the user more accurately. By Contrasting the response time, it could be seen that the distributed TF
-IDF algorithm has a good scalability.

Key words: hadoop, distributed TF-IDF algorithm, mapreduce, textrank algorithm