Abstract: In order to detect the duplicate of Chinese data, we propose a duplicate record detection algorithm based on SNM (Sorted-Neighborhood Method) algorithm, which integrates the extended version of synonym word forest and Chinese word segmentation. Using the extended version of synonym word forest and Jaccard algorithm to calculate the similarity of words, the Chinese word segmentation in Python is used to segment sentences, to optimize cosine similarity algorithm and to calculate the similarity of sentences. The improved algorithm can effectively detect duplicate records of fields and sentences recorded in Chinese. The experiment on the test data set of students in a counseling institution shows that the recall ratio of the new algorithm is much higher than that of the traditional SNM algorithm.

Key words: similar duplicate records, sorted-neighborhood method ( SNM ) algorithm, chinese word segmentation