Lightweight defense mechanism against adversarial attacks via adaptive pruning and robust distillation

Bin WANG; Simin LI; Yaguan QIAN; Jun ZHANG; Chaohao LI; Chenming ZHU; Hongfei ZHANG

doi:10.11959/j.issn.2096-109x.2022074

您当前的位置：

首页 >

文章列表页 >

Lightweight defense mechanism against adversarial attacks via adaptive pruning and robust distillation

Papers and Reports | 更新时间：2024-06-03

- Lightweight defense mechanism against adversarial attacks via adaptive pruning and robust distillation
- Chinese Journal of Network and Information Security Vol. 8, Issue 6, Pages: 102-109(2022)
- 作者机构：
  
  1. 浙江科技学院，浙江杭州 310023
  2. 浙江省多维感知技术应用与安全重点实验室，浙江杭州 310052
  3. 浙江省电子信息产品检验研究院，浙江杭州 310007
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2022074
  CLC： TP393
- Online First：2022-12，
  
  Published：15 December 2022
- 稿件说明：
移动端阅览
Bin WANG, Simin LI, Yaguan QIAN, et al. Lightweight defense mechanism against adversarial attacks via adaptive pruning and robust distillation[J]. Chinese Journal of Network and Information Security, 2022, 8(6): 102-109.
DOI：

Bin WANG, Simin LI, Yaguan QIAN, et al. Lightweight defense mechanism against adversarial attacks via adaptive pruning and robust distillation[J]. Chinese Journal of Network and Information Security, 2022, 8(6): 102-109. DOI： 10.11959/j.issn.2096-109x.2022074.

摘要

对抗训练是一类常用的对抗攻击防御方法，其通过将对抗样本纳入训练过程，从而有效抵御对抗攻击。然而，对抗训练模型的鲁棒性通常依赖于网络容量的提升，即对抗训练所获得的网络为防御对抗攻击而大幅提升网络的模型容量，对其可用性造成较大约束。因此，如何在保证对抗训练模型鲁棒性的同时，降低模型容量，提出轻量对抗攻击防御方法是一大挑战。为解决以上问题，提出一种基于剪枝技术和鲁棒蒸馏融合的轻量对抗攻击防御方法。该方法以对抗鲁棒准确率为优化条件，在对预训练的鲁棒对抗模型进行分层自适应剪枝压缩的基础上，再对剪枝后的网络进行基于数据过滤的鲁棒蒸馏，实现鲁棒对抗训练模型的有效压缩，降低其模型容量。在CIFAR-10和CIFAR-100数据集上对所提出的方法进行性能验证与对比实验，实验结果表明，在相同 TRADES 对抗训练下，所提出的分层自适应剪枝技术相较于现有剪枝技术，其剪枝所得到的网络结构在多种 FLOPs 下均表现出更强的鲁棒性。此外，基于剪枝技术和鲁棒蒸馏融合的轻量对抗攻击防御方法相较于其他鲁棒蒸馏方法表现出更高的对抗鲁棒准确率。因此，实验结果证明所提方法在降低对抗训练模型容量的同时，相较于现有方法具有更强的鲁棒性，提升了对抗训练模型在物联网边缘计算环境的适用性。

Abstract

Adversarial training is one of the commonly used defense methods against adversarial attacks

by incorporating adversarial samples into the training process.However

the effectiveness of adversarial training heavily relied on the size of the trained model.Specially

the size of trained models generated by the adversarial training will significantly increase for defending against adversarial attacks.This imposes constraints on the usability of adversarial training

especially in a resource-constraint environment.Thus

how to reduce the model size while ensuring the robustness of the trained model is a challenge.To address the above issues

a lightweight defense mechanism was proposed against adversarial attacks

with adaptive pruning and robust distillation.A hierarchically adaptive pruning method was applied to the model generated by adversarial training in advance.Then the trained model was further compressed by a modified robust distillation method.Experimental results on CIFAR-10 and CIFAR-100 datasets showed that our hierarchically adaptive pruning method presented stronger robustness under various FLOP than the existing pruning methods.Moreover

the fusion of pruning and robust distillation presented higher robustness than the state-of-art robust distillation methods.Therefore

the experimental results prove that the proposed method can improve the usability of the adversarial training in the IoT edge computing environment.

关键词

Keywords

references

郑远攀 , 李广阳 , 李晔 . 深度学习在图像识别中的应用研究综述 [J ] . 计算机工程与应用 , 2019 , 55 ( 12 ): 20 - 36 .

ZHENG Y P , LI G Y , LI Y . Survey of application of deep learning in image recognition [J ] . Computer Engineering and Applications , 2019 , 55 ( 12 ): 20 - 36 .

鱼昆 , 张绍阳 , 侯佳正 , 等 . 语音识别及端到端技术现状及展望 [J ] . 计算机系统应用 , 2021 , 30 ( 3 ): 14 - 23 .

YU K , ZHANG S Y , HOU J Z , et al . Survey of speech recognition and end-to-end techniques [J ] . Computer Systems ＆ Applications , 2021 , 30 ( 3 ): 14 - 23 .

王睿怡 , 罗森林 , 吴舟婷 , 等 . 深度学习在汉语语义分析的应用与发展趋势 [J ] . 计算机技术与发展 , 2019 , 29 ( 9 ): 110 - 116 .

WANG R Y , LUO S L , WU Z T , et al . Application and development trend of deep learning in Chinese semantic analysis [J ] . Computer Technology and Development , 2019 , 29 ( 9 ): 110 - 116 .

SZEGEDY C , ZAREMBA W , SUTSKEVER I , et al . Intriguing properties of neural networks [C ] // Proceedings of 2nd International Conference on Learning Representations (ICLR 2014) . 2014 .

CARLINI N , WAGNER D . Towards evaluating the robustness of neural networks [C ] // Proceedings of 2017 IEEE Symposium on Security and Privacy (SP) . 2017 .

SONG D , EYKHOLT K , EVTIMOV I , et al . Physical adversarial examples for object detectors [C ] // 12th USENIX Workshop on offensive technologies (WOOT 18) . 2018 .

CARLINI N , WAGNER D . Audio adversarial examples:targeted attacks on speech-to-text [C ] // 2018 IEEE Security and Privacy Workshops (SPW) . 2018 : 1 - 7 .

GOODFELLOW I , SHLENS J , SZEGEDY C . Explaining and harnessing adversarial examples [C ] // Proceedings of 3rd International Conference on Learning Representations (ICLR 2015) . 2015 .

MADRY A , MAKELOV A , SCHMIDT L , et al . Towards deep learning models resistant to adversarial attacks [C ] // Proceedings of 6th International Conference on Learning Representations (ICLR 2018) . 2018 .

GOWAL S , QIN C , UESATO J , et al . Uncovering the limits of adversarial training against norm-bounded adversarial examples [J ] . arXiv:2010.03593 , 2020 .

GOU J , YU B , MAYBANK S J , et al . Knowledge distillation:a survey [J ] . International Journal of Computer Vision , 2021 , 129 ( 6 ): 1789 - 1819 .

PAPERNOT N , MCDANIEL P , WU X , et al . Distillation as a defense to adversarial perturbations against deep neural networks [C ] // IEEE Symposium on Security and Privacy , 2016 : 582 - 597 .

GOLDBLUM M , FOWL L , FEIZI S , et al . Adversarially robust distillation [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . 2020 : 3996 - 4003 .

ZI B , ZHAO S , MA X , et al . Revisiting adversarial robustness distillation:robust soft labels make student better [C ] // Proceedings of 2021 IEEE/CVF International Conference on Computer Vision , 2021 : 16423 - 16432 .

BAI T , LUO J , ZHAO J , et al . Recent advances in adversarial training for adversarial robustness [C ] // Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , 2021 : 4312 - 4321 .

ZHANG H , YU Y , JIAO J , et al . Theoretically principled trade-off between robustness and accuracy [C ] // International Conference on Machine Learning . 2019 : 7472 - 7482 .

ALAYRAC J B , UESATO J , HUANG P S , et al . Are labels required for improving adversarial robustness [C ] // Advances in Neural Information Processing Systems . 2019 : 12192 - 12202 .

CARMON Y , RAGHUNATHAN A , SCHMIDT L , et al . Unlabeled data improves adversarial robustness [C ] // Advances in Neural Information Processing Systems . 2019 : 11190 - 11201 .

SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition [J ] . arXiv:1409.1556 , 2014 .

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C ] // Proceedings of 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2016 : 770 - 778 .

CROCE F , HEIN M . Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks [C ] // Proceedings of the 37th International Conference on Machine Learning . 2020 : 2206 - 2216 .

LI H , KADAV A , DURDANOVIC I , et al . Pruning filters for efficient convnets [C ] // Proceedings of 5th International Conference on Learning Representations (ICLR 2017) . 2017 .

LIU Z , LI J , Shen Z , et al . Learning efficient convolutional networks through network slimming [C ] // Proceedings of 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2017 : 2755 - 2763 .

SUI Y , YIN M , XIE Y , et al . CHIP:channel Independence-based pruning for compact neural networks [C ] // Advances in Neural Information Processing Systems . 2021 : 24604 - 24616 .

Views

494

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Adversarial attacks and defenses in deep learning

Related Author

Ximeng LIU

Lehui XIE

Yaopeng WANG

Xuru LI

Related Institution

College of Mathematics and Computer Science,Fuzhou University

Guangdong Provincial Key Laboratory of Data Security and Privacy Protection

School of Computer Science and Technology,East China Normal University

⁰