Feature selection for Chinese Text Categorization based on improved particle swarm optimization

Abstract

Feature selection is an important preprocessing step of Chinese Text Categorization, which reduces the high dimension and keeps the reduced results comprehensible compared to feature extraction. A novel criterion to filter features coarsely is proposed, which integrating the superiorities of term frequency-inverse document frequency as inner-class measure and CHI-square as inter-class, and a new feature selection method for Chinese text categorization based on swarm intelligence is presented, which using improved particle swarm optimization to select features fine on the results of coarse grain filtering, and utilizing support vector machine to evaluate feature subsets and taking the evaluations as the fitness of particles. The experiments on Fudan University Chinese Text Classification Corpus show a higher classification accuracy obtained by using the new criterion for features filtering and an effective feature reduction ratio attained by utilizing the novel FS method for Chinese text categorization.