Abstract: Aiming at  the problem that the scale of relation extraction model was getting larger and larger, and the time consumption was getting longer and longer, we proposed a knowledge filtering mechanism to construct a lightweight relation extraction model by using the positive soft labels selected. Firstly, knowledge distillation was used to extract knowledge and store knowledge in soft labels. In order to avoid the problem of difficult  absorption of knowledge caused by the large gap between  teachers and  students in knowledge distillation, we used teacher assistant knowledge distillation pattern. Secondly,  the cosine similarity of labels was used to filter the positive soft labels and the positive soft labels were dynamically given  higher weight in each step of the distillation, so as to  weaken the influence caused by  the wrong labels in the knowledge transfer. The experimental results on SemEval-2010 Task 8 dataset show that the proposed  mode can not only complete the task of lightweight relation extraction, but also improve the extraction accuracy.

Key words: lightweight relation extraction, knowledge filtering, positive soft label, knowledge distillation, cosine similarity