Abstract:

In order to solve the problem of imprecision in Chinese webpage theme feature extraction, feature extraction algorithm for Chinese webpage theme is studied. Webpage theme feature extraction is the foundation of topic web crawler to calculate webpage correlation. Considering two classifications of theme webpage, we improved the commonly used text feature item weighting method of TF-IDF(Term Frequency\|Inverse Document Frequency). We combine Semi-structured characteristics of webpage, feature's position information, present a new calculation method of linear feature item weighting. This method can effectively improve the theme webpage recall rate and precision rate.

Key words: term frequency-inverse document frequency(TF-IDF), vector space model, feature, correlation calculation, information gain