Research on adversarial examples defense method based on multi-modal feature fusion

WEI Xuanxuan; LIU Wanping; LU Ling

doi:10.11959/j.issn.2096-109x.2025023

您当前的位置：

首页 >

文章列表页 >

Research on adversarial examples defense method based on multi-modal feature fusion

Papers | 更新时间：2025-05-14

- Research on adversarial examples defense method based on multi-modal feature fusion
- Chinese Journal of Network and Information Security Vol. 11, Issue 2, Pages: 175-188(2025)
- 作者机构：
  
  重庆理工大学计算机科学与工程学院重庆 400054
- 作者简介：
- 基金信息：
  
  The Chongqing Natural Science Foundation(cstc2021jcyj-msxmX0594);Action Plan for High-Quality Development of Graduate Education of Chongqing University of Technology(gzlcx20243171)
- DOI：10.11959/j.issn.2096-109x.2025023
  CLC： TP391
- Received：25 August 2024，
  
  Revised：2025-02-10，
  
  Published：25 April 2025
- 稿件说明：
移动端阅览
魏宣宣, 刘万平, 卢玲. 基于多模态特征融合的对抗样本防御方法研究[J]. 网络与信息安全学报, 2025, 11(2): 175-188.

WEI X X, LIU W P, LU L. Research on adversarial examples defense method based on multi-modal feature fusion[J]. Chinese Journal of Network and Information Security, 2025, 11(2): 175-188.
魏宣宣, 刘万平, 卢玲. 基于多模态特征融合的对抗样本防御方法研究[J]. 网络与信息安全学报, 2025, 11(2): 175-188. DOI： 10.11959/j.issn.2096-109x.2025023.

WEI X X, LIU W P, LU L. Research on adversarial examples defense method based on multi-modal feature fusion[J]. Chinese Journal of Network and Information Security, 2025, 11(2): 175-188. DOI： 10.11959/j.issn.2096-109x.2025023.

摘要

神经网络模型容易受到对抗样本攻击，尤其是在图像分类任务中，对抗攻击会导致其产生错误的分类。为了抵御这种攻击，已经提出多种防御方法。这些防御方法侧重于改进模型结构，或模型仅使用对抗训练方法导致防御类型单一且损害模型分类能力。受人类视觉系统通过多模态感官感知信息的启发，提出将文本描述信息融合到图像信息中的多模态金字塔特征融合（MPFF）防御方法。首先，使用ViT-GPT2模型根据图像信息生成对应的文本描述信息，通过特征金字塔网络获取多尺度信息；然后，利用预训练的TF-IDF模型提取文本描述信息的特征矩阵，利用ResNet50模型提取图像特征；接着，将图像特征和文本特征进行加权融合，得到最终的多模态特征；最后，利用分类器对融合特征进行分类检测。在CIFAR-10和ImageNet数据集上进行了对比实验，实验结果表明，在不同扰动强度的黑盒攻击下，与其他方法相比，所提方法在两个数据集上的准确率分别平均提高21.8%和22.5%。

Abstract

In recent years

the vulnerability of neural network models to adversarial attacks has been a significant concern

particularly in image classification tasks

where such attacks can lead to incorrect classifications. To counteract these attacks

numerous defence methods have been proposed. Existing defence methods have predominantly concentrated on enhancing model structures or adopting adversarial training methods individually

resulting in a single type of defence and potentially compromising the model's classification capability. Drawing on the human visual system's ability to perceive information through multimodal sensory inputs

a multimodal pyramid feature fusion (MPFF) defence method was proposed

integrating textual descriptions of images into the image information. Initially

ViT-GPT2 was utilized to generate corresponding textual descriptions based on image information

while a feature pyramid network captured multi-scale information. Subsequently

a pre-trained TF-IDF model was employed to extract feature matrices from the textual descriptions

and a ResNet50 model was used to extract image features. These image and text features were then weighted and fused to obtain the final multimodal features. Finally

a classifier was applied to perform classification detection using the fused features. Comparative experiments were conducted on the CIFAR-10 and ImageNet datasets. The experimental results demonstrate that the accuracy of the proposed method is improved by 21.8% and 22.5% on average compared to other methods under black-box attacks with varying disturbance intensities on the two datasets respectively.

关键词

Keywords

references

GOODFELLOW I J , SHLENS J , SZEGEDY C . Explaining and harnessing adversarial examples [C ] // Proceeding of the 3rd International Conference on Learning Representations (ICLR 2015). 2015 .

LIU W P , ZHONG S M . Web malware spread modelling and optimal control strategies [J ] . Scientific Reports , 2017 , 7 : 42308 .

SZEGEDY C , ZAREMBA W , SUTSKEVER I , et al . Intriguing properties of neural networks [C ] // Proceeding of International Conference on Learning Representations (ICLR) . 2014 .

PAPERNOT N , MCDANIEL P , WU X , et al . Distillation as a defense to adversarial perturbations against deep neural networks [C ] // Proceedings of the IEEE Symposium on Security and Privacy . Piscataway : IEEE Press , 2016 : 582 - 597 .

XU W , EVANS D , QI Y . Feature squeezing: detecting adversarial examples in deep neural networks [C ] // Proceeding of International Conference on Network and Distributed System Security Symposium . Virginia : the Internet Society , 2018 : 1 - 15 .

WANG D N , CHEN W , YANG Y , et al . Defense method of adversarial training based on gaussian enhancement and iterative attack [J ] . Computer Science , 2021 , 48 ( 6A ): 509 - 513,537 .

VINYALS O , TOSHEV A , BENGIO S , et al . Show and tell: a neural image caption generator [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2015 : 3156 - 3164 .

KURAKIN A , GOODFELLOW I J , BENGIO S . Adversarial examples in the physical world [C ] // Proceedings of the 5th International Conference on Learning Representations . 2018 : 99 - 112 .

CARLINI N , WAGNER D . Towards evaluating the robustness of neural networks [C ] // Proceedings of the IEEE Symposium on Security and Privacy (SP) . Piscataway : IEEE Press , 2017 : 39 - 57 .

MADRY A , MAKELOV A , SCHMIDT L , et al . Towards deep learning models resistant to adversarial attacks [C ] // Proceeding of International Conference on Learning Representations(ICLR) . 2018 .

ILYAS A , ENGSTROM L , ATHALYE A , et al . Black-box adversarial attacks with limited queries and information [C ] // Proceedings of International Conference on Machine Learning . 2018 : 2137 - 2146 .

YUAN Z , ZHANG J , JIANG Z , et al . Adaptive perturbation for adversarial attack [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024 , 46 ( 8 ): 5663 - 5676 .

XIAO C , LI B , ZHU J , et al . Generating adversarial examples with adversa-rial networks [C ] // Proceedings of the 27th International Joint Conference on Artificial Intelligence . 2018 : 3905 - 3911 .

GAUTHIER J . Conditional generative adversarial nets for convolutional face generation [J ] . Class Project for Stanford CS231N: Convolutional Neural Networks for Visual Recognition , Winter Semester, 2014 ( 5 ): 2 .

CHEN J Y , ZHENG H B , CHEN R X , et al . RCA-SOC: a novel adversarial defense by refocusing on critical areas and strengthening object contours [J ] . Computers & Security , 2020 , 96 : 101916 .

ZHU J L , PENG G H , WANG D W . Dual-domain-based adversarial defense with conditional VAE and bayesian network [J ] . IEEE Transactions on Industrial Informatics , 2020 , 17 ( 1 ): 596 - 605 .

DABOUEI A , SOLEYMANI S , TAHERKHANI F , et al . Exploiting joint robustness to adversarial perturbations [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2020 : 1122 - 1131 .

ATHALYE A , CARLINI N , WAGNER D . Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples [C ] // Proceedings of International Conference on Machine Learning . 2018 : 274 - 283 .

PRAKASH A , MORAN N , GARBER S , et al . Protecting JPEG images against adversarial attacks [C ] // Proceedings of Data Compression Conference . Piscataway : IEEE Press , 2018 : 137 - 146 .

SONG Y , KIM T , NOWOZIN S , et al . PixelDefend: leveraging generative models to understand and defend against adversarial examples [C ] // Proceedings of the 6th International Conference on Learning Representations . 2018 .

ZHANG C , GAO P . Countering adversarial examples: combining input transformation and noisy training [C ] // Proceedings of the IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE Press , 2021 : 102 - 111 .

SUN B , TSAI N , LIU F C , et al . Adversarial defense by stratified convolutional sparse coding [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2019 : 11447 - 11456 .

HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network [J ] . Computer Science , 2015 , 14 ( 7 ): 38 - 39 .

肖镇杰 , 黄诗瑀 , 叶锋 , 等 . 基于PatchTracker的对抗补丁防御算法 [J ] . 网络与信息安全学报 , 2024 , 10 ( 1 ): 169 - 180 .

XIAO Z J , HUANG S Y , YE F , et al . Adversarial patch defense algorithm based on PatchTracker [J ] . Chinese Journal of Network and Information Security , 2024 , 10 ( 1 ): 169 - 180 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [C ] // Proceedings of Advances in Neural Information Processing Systems 30(NIPS 2017) . Piscataway : IEEE Press , 2017 : 5998 - 6008 .

HE K M , ZHANG X Y , REN S Q , et al . Deep residual learning for image recognition [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2016 : 770 - 778 .

JIA X , WEI X , CAO X , et al . ComDefend: an efficient image compression model to defend adversarial examples [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2019 : 6084 - 6092 .

LAYKAVIRIYAKUL P , PHAISANGITTISAGUL E . Collaborative defense -GAN for protecting adversarial attacks on classification system [J ] . Expert Systems with Applications , 2023 , 214 : 118957 .

DAI T , FENG Y , WU D X , et al . DIPDefend: deep image prior driven defense against adversarial examples [C ] // Proceedings of the 28th ACM International Conference on Multimedia . New York : ACM Press , 2020 : 1404 - 1412 .

LI X , LI X R , PAN D , et al . Improving adversarial robustness via probabilistically compact loss with logit constraints [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . 2021 : 8482 - 8490 .

ZHANG J H , MAEDA K , OGAWA T , et al . Defense against black-box adversarial attacks via heterogeneous fusion features [C ] // Proceedings of ICASSP 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Piscataway : IEEE Press , 2023 : 1 - 5 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Novel defense based on softmax activation transformation

Study of high-speed malicious Web page detection system based on two-step classifier

Related Author

Jinyin CHEN

Changan WU

Haibin ZHENG

Zheng-qi WANG

Xiao-bing FENG

Chi ZHANG

Related Institution

Institute of Cyberspace Security, Zhejiang University of Technology

College of Information Engineering, Zhejiang University of Technology

University of Science and Technology of China

Key Laboratory of Electromagnetic Space Information, Chinese Academy of Sciences

⁰