Chinese NER based on improved Transformer encoder

Honghao ZHENG; Hongtao YU; Shaomei LI

doi:10.11959/j.issn.2096-109x.2021041

您当前的位置：

首页 >

文章列表页 >

Chinese NER based on improved Transformer encoder

TopicⅡ: Machine Learning and Security Application | 更新时间：2024-06-03

- Chinese NER based on improved Transformer encoder
- Chinese Journal of Network and Information Security Vol. 7, Issue 5, Pages: 105-112(2021)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2021041
  CLC： TP391
- Online First：2021-10，
  
  Published：15 October 2021
- 稿件说明：
移动端阅览
Honghao ZHENG, Hongtao YU, Shaomei LI. Chinese NER based on improved Transformer encoder[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 105-112.
DOI：

Honghao ZHENG, Hongtao YU, Shaomei LI. Chinese NER based on improved Transformer encoder[J]. Chinese Journal of Network and Information Security, 2021, 7(5): 105-112. DOI： 10.11959/j.issn.2096-109x.2021041.

摘要

为了提高中文命名实体识别的效果，提出了基于 XLNET-Transformer_P-CRF 模型的方法，该方法使用了 Transformer_P 编码器，改进了传统 Transformer 编码器不能获取相对位置信息的缺点。实验结果表明，XLNET-Transformer_P-CRF模型在MSRA、OntoNotes4.0、Resume、微博数据集4类数据集上分别达到95.11%、80.54%、96.70%、71.46%的F1值，均高于中文命名实体识别的主流模型。

Abstract

In order to improve the effect of chinese named entity recognition

a method based on the XLNETTransformer_P-CRF model was proposed

which used the Transformer_P encoder

improved the shortcomings of the traditional Transformer encoder that couldn’t obtain relative position information.Experiments show that the XLNET-Transformer_P-CRF model achieves 95.11%

80.54%

96.70%

and 71.46% F1 values on the four types of data sets: MSRA

OntoNotes4.0

Resume

and Weibo

which are all higher than other mainstream chinese NER model.

关键词

Keywords

references

GRISHMAN R , SUNDHEIM B . Message understanding conference-6:a brief history [C ] // International Conference on Computational Linguistics , 1996 : 466 - 471 .

PIZZATO L A , MOLLA D , PARIS C . Pseudo relevance feedback using named entities for question answering [C ] // Proceedings of the 2006 Australian Language Technology Workshop (ALTW-2006) . 2006 : 89 - 90 .

BABYCH B , HARTLEY A . Improving machine translation quality with automatic named entity recognition [C ] // Proceedings of the 7th International EAMT Workshop on MT and other Language Technology Tools,Improving MT Through other Language Technology Tools:Resources and Tools for Building MT,Association for Computational Linguistics . 2003 : 1 - 8 .

MANDL T , WOMSER-HACKER C , . The effect of named entities on effectiveness in cross-language information retrieval evaluation [C ] // Proceedings of the 2005 ACM Symposium on Applied Computing . 2005 : 1059 - 1064 .

邱锡鹏 . 神经网络与深度学习 [M ] . 北京 : 机械工业出版社 , 2020 .

QIU X P . Neural networks and deep learning [M ] . Beijing : China Machine Press , 2020 .

LI J , SUN A , HAN J , et al . A survey on deep learning for named entity recognition [J ] . IEEE Transactions on Knowledge and Data Engineering , 2020 :1.

HAMMERTON J , . Named entity recognition with long short-term memory [C ] // North American Chapter of the Association for Computational Linguistics . 2003 : 172 - 175 .

HUANG Z , XU W , YU K , et al . Bidirectional LSTM-CRF models for sequence tagging [J ] . arXiv:Computation and Language , 2015 .

MA X , HOVY E . End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF [J ] . arXiv:Learning , 2016 .

CHIU J P , NICHOLS E . Named entity recognition with bidirectional LSTM-CNNs [J ] . Transactions of the Association for Computational Linguistics , 2016 , 4 ( 1 ): 357 - 370 .

BENGIO Y , SIMARD P Y , FRASCONI P , et al . Learning long-term dependencies with gradient descent is difficult [J ] . IEEE Transactions on Neural Networks , 1994 , 5 ( 2 ): 157 - 166 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Neural Information Processing Systems . 2017 : 5998 - 6008 .

GUO Q P , QIU X P , LIU P F , et al . Star transformer [C ] // NAACL . 2019 : 1315 - 1325 .

DAI Z , YANG Z , YANG Y , et al . Transformer-XL:attentive language models beyond a fixed-length context [J ] . arXiv:Learning , 2019 .

HUANG A , VASWANI A , USZKOREIT J , et al . Music transformer:generating music with long-term structure [C ] // International Conference on Learning Representations , 2019 .

MIKOLOV T , CHEN K , CORRADO G S , et al . Efficient estimation of word representations in vector space [C ] // International Conference on Learning Representations . 2013 .

杨飘 , 董文永 . 基于BERT嵌入的中文命名实体识别方法 [J ] . 计算机工程 , 2020 , 46 ( 04 ): 40 - 45 , 52 .

YANG P , DONG W Y . Chinese NER based on BERT embedding [J ] . Computer Engineering , 2020 , 46 ( 4 ): 40 - 45 , 52 .

DEVLIN J , CHANG M , LEE K , et al . BERT:pre-training of deep bidirectional transformers for language understanding [J ] . arXiv:Computation and Language , 2018 .

YANG Z , DAI Z , YANG Y , et al . XLNet:generalized autoregressive pretraining for language understanding [J ] . arXiv:Computation and Language , 2019 .

PARIKH A P , TACKSTROM O , DAS D , et al . A decomposable attention model for natural language inference [C ] // Empirical Methods in Natural Language Processing . 2016 : 2249 - 2255 .

YAN H , DENG B , LI X , et al . TENER:adapting transformer encoder for named entity recognition [J ] . arXiv:Computation and Language , 2019 .

SHAW P , USZKOREIT J , VASWANI A , et al . SELF-attention with relative position representations [C ] // North American Chapter of the Association for Computational Linguistics . 2018 : 464 - 468 .

LAFFERTY J , MCCALLUM A , PEREIRA F , et al . Conditional random fields:probabilistic models for segmenting and Labeling Sequence Data [C ] // International Conference on Machine Learning . 2001 : 282 - 289 .

SMITH L N , . Cyclical learning rates for training neural networks [C ] // Workshop on Applications of Computer Vision , 2017 : 464 - 472 .

LEVOW G , . The third international chinese language processing Bakeoff:word segmentation and named entity recognition [C ] // Meeting of the Association for Computational Linguistics . 2006 : 108 - 117 .

Ralph Weischedel . Ontonotes release 4.0 LDC2011T03 [S ] . 2011 .

ZHANG Y , YANG J . Chinese NER using lattice LSTM [C ] // Meeting of the Association for Computational Linguistics . 2018 : 1554 - 1564 .

PENG N , DREDZE M . Named entity recognition for chinese social media with jointly trained embeddings [C ] // Empirical Methods in Natural Language Processing . 2015 : 548 - 554 .

CHEN A , PENG F , SHAN R , et al . Chinese named entity recognition with conditional probabilistic models [C ] // Meeting of the Association for Computational Linguistics . 2006 : 173 - 176 .

ZHANG S , QIN Y , WEN J , et al . Word segmentation and named entity recognition for SIGHAN Bakeoff3 [C ] // Meeting of the Association for Computational Linguistics . 2006 : 158 - 161 .

ZHOU J S , QU W G , ZHANG F . Chinese named entity recognition via joint identification and categorization [J ] . Chinese Journal of Electronics , 2013 , 22 ( 2 ): 225 - 230 .

DONG C H , ZHANG J J , ZONG C Q . Character based LSTM-CRF with radical-level features for Chinese named entity recognition [C ] // International Conference on Computer Processing of Oriental Languages . 2016 : 239 - 250 .

SUI D , CHEN Y , LIU K , et al . Leverage lexical knowledge for chinese named entity recognition via collaborative graph network [C ] // International Joint Conference on Natural Language Processing . 2019 : 3828 - 3838 .

QIU X P , LI X N , YAN H . Flat chinese ner using flat-lattice transformer [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 2020 : 6836 - 6842 .

WANG M , CHE W , MANNING C D , et al . Effective bilingual constraints for semi-supervised learning of named entity recognizers [C ] // National Conference on Artificial Intelligence . 2013 : 919 - 925 .

CHE W , WANG M , MANNING C D , et al . Named entity recognition with bilingual constraints [C ] // North American Chapter of the Association for Computational Linguistics . 2013 : 52 - 62 .

YANG J , ZHANG Y , DONG F , et al . Neural word segmentation with rich pretraining [C ] // Meeting of the Association for Computational Linguistics . 2017 : 839 - 849 .

LIU W , XU T , XU Q , et al . An encoding strategy based word-character LSTM for chinese NER [C ] // North American Chapter of the Association for Computational Linguistics . 2019 : 2379 - 2389 .

GUI T , ZOU Y , ZHANG Q , et al . A lexicon-based graph neural network for chinese NER [C ] // International Joint Conference on Natural Language Processing . 2019 : 1040 - 1050 .

HE H , SUN X . A unified model for cross-domain and semi- supervised named entity recognition in chinese social media [C ] // National Conference on Artificial Intelligence . 2017 : 3216 - 3222 .

Views

455

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰