Contrastive meta-learning framework for few-shot cross-lingual text classification

GUO Jianming; ZHAO Yuran; LIU Gongshen

doi:10.11959/j.issn.2096-109x.2024043

您当前的位置：

首页 >

文章列表页 >

Contrastive meta-learning framework for few-shot cross-lingual text classification

Papers | 更新时间：2024-08-14

- Contrastive meta-learning framework for few-shot cross-lingual text classification
- Chinese Journal of Network and Information Security Vol. 10, Issue 3, Pages: 107-116(2024)
- 作者机构：
  
  上海交通大学网络空间安全学院，上海 200240
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2024043
  CLC： TP393
- Received：09 August 2023，
  
  Revised：2023-12-25，
  
  Published：25 June 2024
- 稿件说明：
移动端阅览
郭建铭, 赵彧然, 刘功申. 用于小样本跨语言文本分类的元对比学习框架[J]. 网络与信息安全学报, 2024, 10(3): 107-116.

GUO J M, ZHAO Y R, LIU G S. Contrastive meta-learning framework for few-shot cross-lingual text classification[J]. Chinese Journal of Network and Information Security, 2024, 10(3): 107-116.
郭建铭, 赵彧然, 刘功申. 用于小样本跨语言文本分类的元对比学习框架[J]. 网络与信息安全学报, 2024, 10(3): 107-116. DOI： 10.11959/j.issn.2096-109x.2024043.

GUO J M, ZHAO Y R, LIU G S. Contrastive meta-learning framework for few-shot cross-lingual text classification[J]. Chinese Journal of Network and Information Security, 2024, 10(3): 107-116. DOI： 10.11959/j.issn.2096-109x.2024043.

摘要

众多的安全风控问题均为文本分类问题，国际场景下的舆情分析等风控问题涉及多种语言，是一大难点。先前的研究表明，通过跨语言语义知识迁移可以显著提高小样本文本分类任务的性能。然而，跨语言文本分类的发展仍面临着一系列挑战。获得语义无关的文本表征是一项困难的任务。不同语言之间的语法结构和句法规则引起文本表征的差异，因此提取通用的语义信息较为困难。此外，跨语言文本分类的标签数据十分稀缺。在很多现实场景中，只能获得少量的标记数据，这严重降低了许多方法的性能。因此需要有效的方式能够在小样本情况下准确地迁移知识，提高分类模型的泛化能力。为应对这些挑战，提出了集成对比学习和元学习的框架，该框架集成了对比学习和元学习的优势，利用对比学习来提取与语言无关的通用语义信息，同时利用元学习快速泛化的优势来改善小样本场景中的知识迁移。此外，提出了基于任务的数据增强方法，以进一步提高所提框架在小样本跨语言文本分类中的性能。通过在两个广泛使用的多语言文本分类数据集上进行大量实验，证实了所提方法能够有效提升文本分类的准确性，可有效应用于风控安全领域。

Abstract

Many security risk control issues

such as public opinion analysis in international scenarios

have been identified as text classification problems

which are challenging due to the involvement of multiple languages. Previous studies have demonstrated that the performance of few-shot text classification tasks can be enhanced through cross-lingual semantic knowledge transfer. However

the advancement of cross-lingual text classification is faced with several challenges. Firstly

it has been found difficult to obtain language-agnostic representations that perform well in cross-lingual transfer. Moreover

the differences in grammatical structure and syntactic rules between different languages cause variations in text representation

making it difficult to extract general semantic information. Additionally

the scarcity of labeled data has been identified as a severe constraint on the performance of most existing methods. In many real-world scenarios

only a small amount of labeled data is available

which has been found to severely degrade the performance of many methods. Therefore

effective methods are needed to accurately transfer knowledge in few-shot situations and improve the generalization ability of classification models. To tackle these challenges

a novel framework was proposed that integrates contrastive learning and meta-learning. Within the framework

contrastive learning was utilized to extract general language-agnostic semantic information

while the rapid generalization advantages of meta-learning were leveraged to improve knowledge transfer in few-shot settings. Furthermore

a task-based data augmentation method was proposed to further improve the performance of the framework in few-shot cross-lingual classification. Extensive experiments conducted on two widely used multilingual text classification datasets show that the proposed method outperforms several strong baselines. This indicates that the method can be effectively applied in the field of risk control and security.

关键词

Keywords

references

PIKULIAK M , ŠIMKO M , BIELIKOVÁ M . Cross-lingual learning for text processing: a survey [J ] . Expert Systems with Applications , 2021 , 165 : 113765 .

HEIJDEN N , YANNAKOUDAKIS H , MISHRA P , et al . Multilingual and cross-lingual document classification: a meta-learning approach [C ] // Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics . 2021 : 1966 - 1976 .

XU Y M , CAO H , DU W Z , et al . A survey of cross-lingual sentiment analysis: methodologies, models and evaluations [J ] . Data Science and Engineering , 2022 , 7 ( 3 ): 279 - 299 .

MINAEE S , KALCHBRENNER N , CAMBRIA E , et al . Deep learning: based text classification [J ] . ACM Computing Surveys , 2022 , 54 ( 3 ): 1 - 40 .

TANEJA K , VASHISHTHA J . Comparison of transfer learning and traditional machine learning approach for text classification [C ] // Proceedings of 2022 9th International Conference on Computing for Sustainable Global Development (INDIACom) . 2022 : 195 - 200 .

LIN Y X , MENG Y X , SUN X F , et al . BertGCN: transductive text classification by combining GNN and BERT [C ] // Proceedings of Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 . 2021 : 1456 - 1462 .

CHEN X L , SUN Y , ATHIWARATKUN B , et al . Adversarial deep averaging networks for cross-lingual sentiment classification [J ] . Transactions of the Association for Computational Linguistics , 2018 , 6 : 557 - 570 .

XU R C , YANG Y M . Cross-lingual distillation for text classification [C ] // Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics . 2017 : 1415 - 1425 .

ALEXIS C , GUILLAUME L . Cross-lingual language model pretraining [C ] // Proceedings of the 33rd International Conference on Neural Information Processing Systems . 2019 : 7059 - 7069 .

JIANG X Z , LIANG Y B , CHEN W Z , et al . XLM-K: improving cross-lingual language model pre-training with multilingual knowledge [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2022 , 36 ( 10 ): 10840 - 10848 .

HUANG L Z , MA D H , LI S J , et al . Text level graph neural network for text classification [C ] // Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . 2019 : 3444 - 3450 .

NOORALAHZADEH F , BEKOULIS G , BJERVA J , et al . Zero-shot cross-lingual transfer with meta learning [C ] // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 2020 : 4547 - 4562 .

CONNEAU A , KRUSZEWSKI G , LAMPLE G , et al . What you can cram into a single vector: probing sentence embeddings for linguistic properties [C ] // Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics . 2018 : 2126 - 2136 .

JAWAHAR G , SAGOT B , SEDDAH D . What does BERT learn about the structure of language [C ] // Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Stroudsburg, PA, USA : Association for Computational Linguistics , 2019 : 3651 - 3657 .

KÖHN A . What's in an embedding? Analyzing word embeddings through multilingual evaluation [C ] // Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing . 2015 : 2067 - 2073 .

CHIA H K , CHUI W C , CHEN P Y . MAML is a noisy contrastive learner in classification [C ] // Proceedings of the Tenth International Conference on Learning Representations . 2022 : 1 - 8 .

NI R , SHU M , SOURI H , et al . The close relationship between contrastive learning and meta-learning [C ] // Proceedings of International Conference on Learning Representations . 2022 : 9 - 16 .

FINN C , ABBEEL P , LEVINE S . Model-agnostic meta-learning for fast adaptation of deep networks [C ] // Proceedings of the 34th International Conference on Machine Learning . 2017 : 1126 - 1135 .

CHEN T , KORNBLITH S , NOROUZI M , et al . A simple framework for contrastive learning of visual representations [C ] // Proceedings of the 37th International Conference on Machine Learning . 2020 : 1597 - 1607 .

TIAN Y L , SUN C , POOLE B , et al . What makes for good views for contrastive learning [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . 2020 : 6827 - 6839 .

YOU Y N , CHEN T L , SUI Y D , et al . Graph contrastive learning with augmentations [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . 2020 : 5812 - 5823 .

JANARTHANAN R , ALEX I , ERIC J . Meta-learning requires meta-augmentation [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . 2020 : 5705 - 5715 .

YAO H , HUANG L , ZHANG L , et al . Improving generalization in meta-learning via task augmentation [C ] // International Conference on Machine Learning , ICML . 2020 : 1 - 8 .

JACOB D , MING-WEI C , KENTON L , et al . BERT: pre-training of deep bidirectional transformers for language understanding [C ] // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics . 2019 : 4171 - 4186 .

CONNEAU A , KHANDELWAL K , GOYAL N , et al . Unsupervised cross-lingual representation learning at scale [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 2020 : 8440 - 8451 .

LIU Y H , GU J T , GOYAL N , et al . Multilingual denoising pre-training for neural machine translation [J ] . Transactions of the Association for Computational Linguistics , 2020 , 8 : 726 - 742 .

ROUDITCHENKO A , CHUANG Y S , SHVETSOVA N , et al . C2KD: cross-lingual cross-modal knowledge distillation for multilingual text-video retrieval [C ] // Proceedings of ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 2023 : 1 - 5 .

WANG Z Y , LIU X , YANG P J , et al . Cross-lingual text classification with heterogeneous graph neural network [C ] // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing . 2021 : 612 - 620 .

ZHANG J Y , LIANG T , WAN M Y , et al . Curriculum knowledge distillation for emoji-supervised cross-lingual sentiment analysis [C ] // Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . 2022 : 864 - 875 .

GHOSH S , MAJI S , DESARKAR M S . GNoM: graph neural network enhanced language models for disaster related multilingual text classification [C ] // Proceedings of the 14th ACM Web Science Conference 2022 . 2022 : 55 - 65 .

VAN DER HEIJDEN N , SHUTOVA E , YANNAKOUDAKIS H . K-hop neighborhood regularization for few-shot learning on graphs: a case study of text classification [C ] // Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics . 2023 : 1187 - 1200 .

JAKE S , KEVIN S , RICHARD Z . Prototypical networks for few-shot learning [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS , 2017 : 4080 – 4090 .

ALEX N . On first-order meta-learning algo-rithms [EB ] . arXiv Preprint. arXiv: 1803 . 02999 ( 2018 ).

ELENI T , ZHU T , VINCENT D , et al . Meta-Dataset: a dataset of datasets for learning to learn from few examples [C ] // Proceedings of International Conference on Learning Representations . 2020 : 1 - 8 .

HEIKE A , NGOC T , TANJA S . Combination of recurrent neural networks and factored language models for code-switching language modeling [C ] // Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics . 2013 : 206 - 211 .

CHEN J A , TAM D , RAFFEL C , et al . An empirical survey of data augmentation for limited data learning in NLP [J ] . Transactions of the Association for Computational Linguistics , 2023 , 11 : 191 - 211 .

LI Y , FUNG P . Improved mixed language speech recognition using asymmetric acoustic model and language model with code-switch inversion constraints [C ] // Proceedings of 2013 IEEE International Conference on Acoustics, Speech and Signal Processing . 2013 : 7368 - 7372 .

YANG J , MA S M , ZHANG D D , et al . Alternating language modeling for cross-lingual pre-training [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2020 , 34 ( 5 ): 9386 - 9393 .

VERMA V , LAMB A , BECKHAM C , et al . Manifold mixup: better representations by interpolating hidden states [C ] // Proceedings of the 36th International Conference on Machine Learning , 2019 : 6438 - 6447 .

QIN L B , NI M H , ZHANG Y , et al . CoSDA-ML: multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP [C ] // Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence . California : International Joint Conferences on Artificial Intelligence Organization . 2020 : 3853 - 3860 .

KRISHNAN J , ANASTASOPOULOS A , PUROHIT H , et al . Multilingual code-switching for zero-shot cross-lingual intent prediction and slot filling [C ] // Proceedings of the 1st Workshop on Multilingual Representation Learning . 2021 : 211 - 223 .

FENG Y K , LI F , KOEHN P . Toward the limitation of code-switching in cross-lingual transfer [C ] // Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . 2022 : 5966 - 5971 .

HARDALOV M , ARORA A , NAKOV P , et al . Few-shot cross-lingual stance detection with sentiment-based pre-training [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2022 , 36 ( 10 ): 10729 - 10737 .

HOLGER S , LI X . A corpus for multilingual document classification in eight languages [C ] // Proceedings of the Eleventh International Conference on Language Resources and Evaluation . 2018 .

PRETTENHOFER P , STEIN B . Cross-language text classification using structural correspondence learning [C ] // Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics . 2010 : 1118 - 1127 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Facial recognition targeted attack algorithm based on multiscale frequency decomposition and meta-learning

Human action recognition method based on multi-view semi-supervised ensemble learning

Related Author

CAI Jun

HUANG Tianqiang

ZHENG Aokun

YE Feng

XU Chao

Shengnan CHEN

Xinmin FAN

Related Institution

College of Computer and Cyber Security, Fujian Normal University

Digital Fujian Institute of Big Data Security Technology

Fujian Normal University

Fujian Engineering Research Center, Network Security and Educational Informatization

⁰