mmDefender: enhancing large language model with multimodal multilingual adversarial training

ZHANG Daojuan; ZHAO Yufei; ZHANG Peng; WU Tianqi; CHEN Kai; DAI Yuwen

doi:10.11959/j.issn.2096-109x.2025049

您当前的位置：

首页 >

文章列表页 >

mmDefender: enhancing large language model with multimodal multilingual adversarial training

Papers | 更新时间：2025-09-12

- mmDefender: enhancing large language model with multimodal multilingual adversarial training
- Chinese Journal of Network and Information Security Vol. 11, Issue 4, Pages: 160-172(2025)
- 作者机构：
  
  中国电力科学研究院有限公司电力网络安全防护与监测技术实验室，北京 102209
- 作者简介：
- 基金信息：
  
  Science and Technology Project of State Grid Corporation of China(5700-202358708A-3-3-JC)
- DOI：10.11959/j.issn.2096-109x.2025049
  CLC： TP393
- Received：29 April 2025，
  
  Revised：2025-06-13，
  
  Published：25 August 2025
- 稿件说明：
移动端阅览
张道娟, 赵宇飞, 张錋, 等. mmDefender: 基于多模态多语言对抗训练的大语言模型增强方法[J]. 网络与信息安全学报, 2025, 11(4): 160-172.

Format: ZHANG D J, ZHAO Y F, ZHANG P, et al. mmDefender: enhancing large language model with multimodal multilingual adversarial training[J]. Chinese Journal of Network and Information Security, 2025, 11(4): 160-172.
张道娟, 赵宇飞, 张錋, 等. mmDefender: 基于多模态多语言对抗训练的大语言模型增强方法[J]. 网络与信息安全学报, 2025, 11(4): 160-172. DOI： 10.11959/j.issn.2096-109x.2025049.

Format: ZHANG D J, ZHAO Y F, ZHANG P, et al. mmDefender: enhancing large language model with multimodal multilingual adversarial training[J]. Chinese Journal of Network and Information Security, 2025, 11(4): 160-172. DOI： 10.11959/j.issn.2096-109x.2025049.

摘要

多模态多语言文本生成作为自然语言处理的前沿领域，通过融合视觉与文本输入增强跨语言内容生成能力。这种方法突破了传统纯文本模型的限制，将视觉线索纳入文本生成过程，从而实现更丰富且可以上下文感知的语言细微差别理解。首先探讨了多模态元素融入多语言文本生成系统时引入的挑战与脆弱性，重点研究对抗攻击与数据投毒如何通过操纵视觉上下文来影响生成翻译的准确性与可靠性。为应对这些威胁，提出一种基于噪声对比学习的新型防御方法。该方法通过训练系统区分真实视觉输入与篡改内容，显著增强了系统的鲁棒性，有效缓解恶意攻击对翻译过程的干扰。该研究不仅揭示了多模态数据在多语言翻译架构中的潜在风险，还为构建安全、可靠的跨模态生成系统提供了切实可行的解决方案，对推动人工智能模型在实际场景中的安全落地具有重要意义。

Abstract

Multilingual and multimodal text generation was recognized as a cutting-edge field in natural language processing

in which cross-lingual content generation was enhanced through the integration of visual and textual inputs. By incorporating visual cues into the text generation process

the limitations of traditional text-only models were overcome

enabling a richer and more context-aware understanding of linguistic nuances. The challenges and vulnerabilities introduced by multimodal elements in multilingual text generation systems were systematically investigated

with a focus on how adversarial attacks and data poisoning could manipulate visual context to compromise translation accuracy and reliability. To address these threats

a novel defense method based on noise contrastive learning was proposed. The system was trained to distinguish genuine visual inputs from tampered content

which significantly improved its robustness and effectively mitigated the interference of malicious attacks on the translation process. This study not only revealed the potential risks posed by multimodal data in multilingual translation architectures

but also provided practical solutions for building secure and reliable cross-modal generation systems. These contributions are crucial for ensuring the safe deployment of AI models in real-world applications.

关键词

Keywords

references

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 . 2017 : 5998 - 6008 .

TOUVRON H , LAVRIL T , IZACARD G , et al . LLaMA: open and efficient foundation language models [EB ] . 2023 .

LIU Z L , LI Y W , SHU P , et al . Radiology-LLaMA2: best-in-class large language model for radiology [EB ] . 2023 .

XHONNEUX S , SORDONI A , GÜNNEMANN S , et al . Efficient adversarial training in LLMs with continuous attacks [EB ] . 2024 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [C ] // Proceedings of the 38th International Conference on Machine Learning . New York : PMLR , 2021 : 8738 - 8753 .

XU K , BA J , KIROS R , et al . Show, attend and tell: neural image caption generation with visual attention [EB ] . 2015 .

ANTOL S , AGRAWAL A , LU J S , et al . VQA: visual question answering [C ] // Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2015 : 2425 - 2433 .

RAMESH A , PAVLOV M , GOH G , et al . Zero-shot text-to-image generation [C ] // Proceedings of International Conference on Machine Learning . New York : PMLR , 2021 : 8821 - 8831 .

魏宣宣 , 刘万平 , 卢玲 . 基于多模态特征融合的对抗样本防御方法研究 [J ] . 网络与信息安全学报 , 2025 , 11 ( 2 ): 175 - 188 .

WEI X X , LIU W P , LU L . Research on adversarial examples defense method based on multi-modal feature fusion [J ] . Chinese Journal of Network and Information Security , 2025 , 11 ( 2 ): 175 - 188 .

田梓琳 , 龙云飞 , 田野 , 等 . 针对音频分类的跨表征可迁移对抗样本生成方法 [J ] . 网络与信息安全学报 , 2025 , 11 ( 2 ): 125 - 135 .

TIAN Z L , LONG Y F , TIAN Y , et al . Cross-representation transferable adversarial examples generation for audio classification [J ] . Chinese Journal of Network and Information Security , 2025 , 11 ( 2 ): 125 - 135 .

LIU Y H , YE H T , WEISSWEILER L , et al . A crosslingual investigation of conceptualization in 1335 languages [EB ] . 2023 .

ZHANG B , BAPNA A , JOHNSON M , et al . Multilingual document-level translation enables zero-shot transfer from sentences to documents [C ] // Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2022 : 4176 - 4192 .

KSHETRI N , ALCANTARA L L . Cyber-threats and cybersecurity challenges [M ] . London : Routledge , 2015 : 285 - 293 .

JI H Y , YANG J , CHAI L Z , et al . SEvenLLM: benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence [EB ] . 2024 .

OpenAI . Improving robustness of large language models via inference - time enhancement [EB ] . 2025 .

FAN X J , TAO C L . Towards resilient and efficient LLMs: a comparative study of efficiency, performance, and adversarial robustness [EB ] . 2024 .

YU L , ZHANG H Y , XU C S . Text-guided attention is all you need for zero-shot robustness in vision-language models [EB ] . 2024 .

CONNEAU A , KHANDELWAL K , GOYAL N , et al . Unsupervised cross-lingual representation learning at scale [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2020 : 8440 - 8451 .

VAN-DEN-OORD A , LI Y Z , VINYALS O , et al . Representation learning with contrastive predictive coding [EB ] . 2018 .

ELLIOTT D , FRANK S , SIMA’AN K , et al . Multi30K: multilingual english-german image descriptions [C ] // Proceedings of the 5th Workshop on Vision and Language . Stroudsburg : ACL , 2016 : 70 - 74 .

LIN T Y , MAIRE M , BELONGIE S , et al . Microsoft COCO: common objects in context [M ] // Computer Vision – ECCV 2014 . Cham : Springer International Publishing , 2014 : 740 - 755 .

BARRAULT L , BOUGARES F , SPECIA L , et al . Findings of the third shared task on multimodal machine translation [C ] // Proceedings of the Third Conference on Machine Translation: Shared Task Papers . Stroudsburg : ACL , 2018 : 304 - 323 .

YANG J , GUO H , YIN Y , et al . M 3 P: towards multimodal multilingual translation with multimodal prompt [C ] // Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation . Torino : ELRA and ICCL , 2024 : 10858 - 10871 .

CHENG X X , ZHU Z H , LI Y W , et al . DAS-CL: towards multimodal machine translation via dual-level asymmetric contrastive learning [C ] // Proceedings of the 32nd ACM International Conference on Information and Knowledge Management . New York : ACM , 2023 : 337 - 347 .

KINGMA D P , BA J . Adam: a method for stochastic optimization [C ] // Proceedings of the 3rd International Conference on Learning Representations . 2015 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

AutoVulnFix: LLM-driven intelligent search and adaptive vulnerability repair system

Bidirectional privacy authorization for large models based on private set intersection

Survey on automated vulnerability mining techniques for IoT device firmware

Detecting privacy compliance of mobile applications from the perspective of the "minimum necessary" principle

Instruction and demonstration-based secure service attribute generation mechanism for textual data

Related Author

Han Hao

Wu Zicheng

Zhao Xinjian

Ding Yixin

WANG Yuanyuan

FENG Haokang

WU Xuguang

HAN Yiliang

Related Institution

State Grid Jiangsu Electric Power Co., Ltd.

Nanjing University of Aeronautics and Astronautics

College of Cryptography Engineering, Engineering University of PAP

Key Laboratory of Counter-Terrorism Command Information Engineering of the Ministry of Education, Engineering University of PAP

Information Engineering University

⁰