mmDefender: 基于多模态多语言对抗训练的大语言模型增强方法

张道娟; 赵宇飞; 张錋; 吴天琦; 陈凯; 戴聿雯

doi:10.11959/j.issn.2096-109x.2025049

您当前的位置：

首页 >

文章列表页 >

mmDefender: 基于多模态多语言对抗训练的大语言模型增强方法

学术论文 | 更新时间：2025-09-12

- mmDefender: 基于多模态多语言对抗训练的大语言模型增强方法
- mmDefender: enhancing large language model with multimodal multilingual adversarial training
- 网络与信息安全学报 2025年11卷第4期页码：160-172
- 作者机构：
  
  中国电力科学研究院有限公司电力网络安全防护与监测技术实验室，北京 102209
- 作者简介：
  
  [ "张道娟（1989- ），女，山东临沂人，中国电力科学研究院有限公司高级工程师，主要研究方向为人工智能安全、智能攻防和威胁检测。" ]
  [ "赵宇飞（1990- ），男，山西太原人，中国电力科学研究院有限公司中级工程师，主要研究方向为网络流量分析、软件逆向工程和人工智能安全。" ]
  [ "张錋（1981- ），男，北京人，中国电力科学研究院有限公司高级工程师，主要研究方向为人工智能安全、智能攻防和威胁检测。" ]
  [ "吴天琦（1996- ），女，河南三门峡人，中国电力科学研究院有限公司中级工程师，主要研究方向为工控系统安全、人工智能安全。" ]
  陈凯（1985- ），男，天津人，中国电力科学研究院有限公司高级工程师，主要研究方向为工控系统安全、人工智能安全。
  [ "戴聿雯（1982- ），女，安徽合肥人，中国电力科学研究院有限公司高级工程师，主要研究方向为智能攻防、人工智能安全。" ]
- 基金信息：
  
  国家电网有限公司总部科技项目(5700-202358708A-3-3-JC)
- DOI：10.11959/j.issn.2096-109x.2025049
  中图分类号： TP393
- 收稿：2025-04-29，
  
  修回：2025-06-13，
  
  纸质出版：2025-08-25
- 稿件说明：
移动端阅览
张道娟, 赵宇飞, 张錋, 等. mmDefender: 基于多模态多语言对抗训练的大语言模型增强方法[J]. 网络与信息安全学报, 2025, 11(4): 160-172.

Format: ZHANG D J, ZHAO Y F, ZHANG P, et al. mmDefender: enhancing large language model with multimodal multilingual adversarial training[J]. Chinese Journal of Network and Information Security, 2025, 11(4): 160-172.
张道娟, 赵宇飞, 张錋, 等. mmDefender: 基于多模态多语言对抗训练的大语言模型增强方法[J]. 网络与信息安全学报, 2025, 11(4): 160-172. DOI： 10.11959/j.issn.2096-109x.2025049.

Format: ZHANG D J, ZHAO Y F, ZHANG P, et al. mmDefender: enhancing large language model with multimodal multilingual adversarial training[J]. Chinese Journal of Network and Information Security, 2025, 11(4): 160-172. DOI： 10.11959/j.issn.2096-109x.2025049.

摘要

多模态多语言文本生成作为自然语言处理的前沿领域，通过融合视觉与文本输入增强跨语言内容生成能力。这种方法突破了传统纯文本模型的限制，将视觉线索纳入文本生成过程，从而实现更丰富且可以上下文感知的语言细微差别理解。首先探讨了多模态元素融入多语言文本生成系统时引入的挑战与脆弱性，重点研究对抗攻击与数据投毒如何通过操纵视觉上下文来影响生成翻译的准确性与可靠性。为应对这些威胁，提出一种基于噪声对比学习的新型防御方法。该方法通过训练系统区分真实视觉输入与篡改内容，显著增强了系统的鲁棒性，有效缓解恶意攻击对翻译过程的干扰。该研究不仅揭示了多模态数据在多语言翻译架构中的潜在风险，还为构建安全、可靠的跨模态生成系统提供了切实可行的解决方案，对推动人工智能模型在实际场景中的安全落地具有重要意义。

Abstract

Multilingual and multimodal text generation was recognized as a cutting-edge field in natural language processing

in which cross-lingual content generation was enhanced through the integration of visual and textual inputs. By incorporating visual cues into the text generation process

the limitations of traditional text-only models were overcome

enabling a richer and more context-aware understanding of linguistic nuances. The challenges and vulnerabilities introduced by multimodal elements in multilingual text generation systems were systematically investigated

with a focus on how adversarial attacks and data poisoning could manipulate visual context to compromise translation accuracy and reliability. To address these threats

a novel defense method based on noise contrastive learning was proposed. The system was trained to distinguish genuine visual inputs from tampered content

which significantly improved its robustness and effectively mitigated the interference of malicious attacks on the translation process. This study not only revealed the potential risks posed by multimodal data in multilingual translation architectures

but also provided practical solutions for building secure and reliable cross-modal generation systems. These contributions are crucial for ensuring the safe deployment of AI models in real-world applications.

关键词

Keywords

references

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 . 2017 : 5998 - 6008 .

TOUVRON H , LAVRIL T , IZACARD G , et al . LLaMA: open and efficient foundation language models [EB ] . 2023 .

LIU Z L , LI Y W , SHU P , et al . Radiology-LLaMA2: best-in-class large language model for radiology [EB ] . 2023 .

XHONNEUX S , SORDONI A , GÜNNEMANN S , et al . Efficient adversarial training in LLMs with continuous attacks [EB ] . 2024 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [C ] // Proceedings of the 38th International Conference on Machine Learning . New York : PMLR , 2021 : 8738 - 8753 .

XU K , BA J , KIROS R , et al . Show, attend and tell: neural image caption generation with visual attention [EB ] . 2015 .

ANTOL S , AGRAWAL A , LU J S , et al . VQA: visual question answering [C ] // Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2015 : 2425 - 2433 .

RAMESH A , PAVLOV M , GOH G , et al . Zero-shot text-to-image generation [C ] // Proceedings of International Conference on Machine Learning . New York : PMLR , 2021 : 8821 - 8831 .

魏宣宣 , 刘万平 , 卢玲 . 基于多模态特征融合的对抗样本防御方法研究 [J ] . 网络与信息安全学报 , 2025 , 11 ( 2 ): 175 - 188 .

WEI X X , LIU W P , LU L . Research on adversarial examples defense method based on multi-modal feature fusion [J ] . Chinese Journal of Network and Information Security , 2025 , 11 ( 2 ): 175 - 188 .

田梓琳 , 龙云飞 , 田野 , 等 . 针对音频分类的跨表征可迁移对抗样本生成方法 [J ] . 网络与信息安全学报 , 2025 , 11 ( 2 ): 125 - 135 .

TIAN Z L , LONG Y F , TIAN Y , et al . Cross-representation transferable adversarial examples generation for audio classification [J ] . Chinese Journal of Network and Information Security , 2025 , 11 ( 2 ): 125 - 135 .

LIU Y H , YE H T , WEISSWEILER L , et al . A crosslingual investigation of conceptualization in 1335 languages [EB ] . 2023 .

ZHANG B , BAPNA A , JOHNSON M , et al . Multilingual document-level translation enables zero-shot transfer from sentences to documents [C ] // Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2022 : 4176 - 4192 .

KSHETRI N , ALCANTARA L L . Cyber-threats and cybersecurity challenges [M ] . London : Routledge , 2015 : 285 - 293 .

JI H Y , YANG J , CHAI L Z , et al . SEvenLLM: benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence [EB ] . 2024 .

OpenAI . Improving robustness of large language models via inference - time enhancement [EB ] . 2025 .

FAN X J , TAO C L . Towards resilient and efficient LLMs: a comparative study of efficiency, performance, and adversarial robustness [EB ] . 2024 .

YU L , ZHANG H Y , XU C S . Text-guided attention is all you need for zero-shot robustness in vision-language models [EB ] . 2024 .

CONNEAU A , KHANDELWAL K , GOYAL N , et al . Unsupervised cross-lingual representation learning at scale [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Stroudsburg : ACL , 2020 : 8440 - 8451 .

VAN-DEN-OORD A , LI Y Z , VINYALS O , et al . Representation learning with contrastive predictive coding [EB ] . 2018 .

ELLIOTT D , FRANK S , SIMA’AN K , et al . Multi30K: multilingual english-german image descriptions [C ] // Proceedings of the 5th Workshop on Vision and Language . Stroudsburg : ACL , 2016 : 70 - 74 .

LIN T Y , MAIRE M , BELONGIE S , et al . Microsoft COCO: common objects in context [M ] // Computer Vision – ECCV 2014 . Cham : Springer International Publishing , 2014 : 740 - 755 .

BARRAULT L , BOUGARES F , SPECIA L , et al . Findings of the third shared task on multimodal machine translation [C ] // Proceedings of the Third Conference on Machine Translation: Shared Task Papers . Stroudsburg : ACL , 2018 : 304 - 323 .

YANG J , GUO H , YIN Y , et al . M 3 P: towards multimodal multilingual translation with multimodal prompt [C ] // Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation . Torino : ELRA and ICCL , 2024 : 10858 - 10871 .

CHENG X X , ZHU Z H , LI Y W , et al . DAS-CL: towards multimodal machine translation via dual-level asymmetric contrastive learning [C ] // Proceedings of the 32nd ACM International Conference on Information and Knowledge Management . New York : ACM , 2023 : 337 - 347 .

KINGMA D P , BA J . Adam: a method for stochastic optimization [C ] // Proceedings of the 3rd International Conference on Learning Representations . 2015 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

AutoVulnFix: 大语言模型驱动的智能搜索自适应漏洞修复系统

基于隐私集合求交的大模型双向隐私授权

物联网设备固件自动化漏洞挖掘技术研究综述

“最小必要”原则视角检测移动应用程序的隐私合规性

基于指令与描述的文本数据安全业务属性生成机制