Abstract:

On the basis of the analysis of the process of dealing with the Chinese word segmentation ambiguity, this paper covers bidirectional sc
an word segmentation algorithm based on the context. In order to improve the word segmentation dictionary, the authors put the fixed phrase into the dictionary and discussed the feature selection and the weighting schema enactment in detail. In order to solve the problem of general TFIDF weighting schema at present, we took statistics into consideration, and meanwhile put up the itemscoring method which improves the efficiency of the feature item about text categorization. At last we proved the advantage of the improved weighting schema through test.

Key words: text categorization； context bidirectional scan； vector space model； weighting schema； feature selection