关键词: 文本分类, BERT 预训练模型, 双向门控循环单元, 卷积神经网络 

Abstract: To address the problem that traditional language models can not solve the problem of deep bidirectional representation and the problem that classification models can not adequately capture salient features of text, a text classification model based on BERT-BiGRU-CNN ( Bidirectional Encoder Representation from Transformers-Bidirectional Gating Recurrent Unit-Convolutional Neural Networks) is proposed. Firstly, the BERT pre-training model is used for text representation; secondly, the output data of BERT is input into BiGRU to capture the global semantic information of text. The results of BiGRU layer again are input into CNN to capture the local semantic features of text. Finally, the feature vectors are input into Softmax layer to obtain the classification results. The Chinese news text headlines dataset is used, and the experimental results show that the BERT-BiGRU-CNN based text classification model achieves an F1 value of 0. 948 5 on the dataset, which is better than other baseline models, proving that the BERT-BiGRU-CNN model can improve theshort text classification performance. 

Key words: text classification, bidirectional encoder representation from transformers(BERT)word embedding, bidirectional gating recurrent unit(BiGRU), convolutional neural networks(CNN)