人工智能模型数据泄露的攻击与防御研究综述

任奎; 孟泉润; 闫守琨; 秦湛

doi:10.11959/j.issn.2096-109x.2021001

您当前的位置：

首页 >

文章列表页 >

人工智能模型数据泄露的攻击与防御研究综述

综述 | 更新时间：2024-06-03

- 人工智能模型数据泄露的攻击与防御研究综述
- Survey of artificial intelligence data security and privacy protection
- 网络与信息安全学报 2021年7卷第1期页码：1-10
- 作者机构：
- 作者简介：
  
  [ "任奎（1978- ），男，安徽芜湖人，浙江大学教授、博士生导师，主要研究方向为人工智能安全、数据安全、物联网安全。" ]
  [ "孟泉润（1994- ），男，河南新乡人，浙江大学硕士生，主要研究方向为数据安全与隐私保护。" ]
  [ "闫守琨（1996- ），男，辽宁大连人，浙江大学硕士生，主要研究方向为人工智能安全与对抗攻防。" ]
  [ "秦湛（1988- ），男，北京人，浙江大学研究员、博士生导师，主要研究方向为数据安全与隐私保护、人工智能安全。" ]
- 基金信息：
  
  科技创新2030——“新一代人工智能”重大项目;The National Key Research and Development Project(2020AAA0107700)
- DOI：10.11959/j.issn.2096-109x.2021001
  中图分类号： TP393
- 网络首发：2021-02，
  
  纸质出版：2021-02-15
- 稿件说明：
移动端阅览
任奎, 孟泉润, 闫守琨, 等. 人工智能模型数据泄露的攻击与防御研究综述[J]. 网络与信息安全学报, 2021,7(1):1-10.

Kui REN, Quanrun MENG, Shoukun YAN, et al. Survey of artificial intelligence data security and privacy protection[J]. Chinese Journal of Network and Information Security, 2021, 7(1): 1-10.
任奎, 孟泉润, 闫守琨, 等. 人工智能模型数据泄露的攻击与防御研究综述[J]. 网络与信息安全学报, 2021,7(1):1-10. DOI： 10.11959/j.issn.2096-109x.2021001.

Kui REN, Quanrun MENG, Shoukun YAN, et al. Survey of artificial intelligence data security and privacy protection[J]. Chinese Journal of Network and Information Security, 2021, 7(1): 1-10. DOI： 10.11959/j.issn.2096-109x.2021001.

摘要

人工智能和深度学习算法正在高速发展，这些新兴技术在音视频识别、自然语言处理等领域已经得到了广泛应用。然而，近年来研究者发现，当前主流的人工智能模型中存在着诸多安全隐患，并且这些隐患会限制人工智能技术的进一步发展。因此，研究了人工智能模型中的数据安全与隐私保护问题。对于数据与隐私泄露问题，主要研究了基于模型输出的数据泄露问题和基于模型更新的数据泄露问题。在基于模型输出的数据泄露问题中，主要探讨了模型窃取攻击、模型逆向攻击、成员推断攻击的原理和研究现状；在基于模型更新的数据泄露问题中，探讨了在分布式训练过程中，攻击者如何窃取隐私数据的相关研究。对于数据与隐私保护问题，主要研究了常用的3类防御方法，即模型结构防御，信息混淆防御，查询控制防御。综上，围绕人工智能深度学习模型的数据安全与隐私保护领域中最前沿的研究成果，探讨了人工智能深度学习模型的数据窃取和防御技术的理论基础、重要成果以及相关应用。

Abstract

Artificial intelligence and deep learning algorithms are developing rapidly.These emerging techniques have been widely used in audio and video recognition

natural language processing and other fields.However

in recent years

researchers have found that there are many security risks in the current mainstream artificial intelligence model

and these problems will limit the development of AI.Therefore

the data security and privacy protection was studied in AI.For data and privacy leakage

the model output based and model update based problem of data leakage were studied.In the model output based problem of data leakage

the principles and research status of model extraction attack

model inversion attack and membership inference attack were discussed.In the model update based problem of data leakage

how attackers steal private data in the process of distributed training was discussed.For data and privacy protection

three kinds of defense methods

namely model structure defense

information confusion defense and query control defense were studied.In summarize

the theoretical foundations

classic algorithms of data inference attack techniques were introduced.A few research efforts on the defense techniques were described in order to provoke further research efforts in this critical area.

关键词

Keywords

references

ATENIESE G , MANCINI L V , SPOGNARDI A , et al . Hacking smart machines with smarter ones:how to extract meaningful data from machine learning classifiers [J ] . International Journal of Security and Networks , 2015 , 10 ( 3 ): 137 - 150 .

JUUTI M , SZYLLER S , MARCHAL S , et al . PRADA:protecting against DNN model stealing attacks [C ] // In IEEE European Symposium on Security and Privacy . 2019 : 512 - 527 .

YANG Q , LIU Y , CHEN T , et al . Federated machine learning:concept and applications [J ] . ACM Transactions on Intelligent Systems and Technology (TIST) , 2019 , 10 ( 2 ): 1 - 19 .

PAPERNOT N , MCDANIEL P D , GOODFELLOW I J , et al . Practical black-box attacks against machine learning [C ] // In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security . 2017 : 506 - 519 .

TRAMÈR F , ZHANG F , JUELS A , et al . Stealing machine learning models via prediction APIs [C ] // In 25th USENIX Security Symposium,USENIX Security 16 . 2016 : 601 - 618 .

WANG B H , GONG N Z . Stealing hyperparameters in machine learning [C ] // In 2018 IEEE Symposium on Security and Privacy . 2018 : 36 - 52 .

OH S J , SCHIELE B , FRITZ M . Towards reverse-engineering black-box neural networks [J ] . arXiv:1711.01768 , 2019 .

SATHISH K , RAMASUBBAREDDY S , GOVINDA K . Detection and localization of multiple objects using VGGNet and single shot detection [M ] // Emerging Research in Data Engineering Systems and Computer Communications.Singapore:Springer . 2020 : 427 - 439 .

TARG S , ALMEIDA D , LYMAN K . Resnet in resnet:generalizing residual architectures [J ] . arXiv preprint arXiv:1603.08029 , 2016 .

CORREIA-SILVA J R , BERRIEL R F , BADUE C , et al . Copycat CNN:stealing knowledge by persuading confession with random non-labeled data [C ] // In 2018 International Joint Conference on Neural Networks . 2018 : 1 - 8 .

BATINA L,BHASINS , JAP D , et al . CSI NN:reverse engineering of neural network architectures through electromagnetic side channel [C ] // In 28th USENIX Security Symposium,USENIX Security 2019 . 2019 : 515 - 532 .

YU H G , YANG K C , ZHANG T , et al . Cloudleak:large-scale deep learning models stealing through adversarial examples [C ] // Network and Distributed System Security Symposium . 2020 .

FREDRIKSON M , JHA S , RISTENPART T . Model inversion attacks that exploit confidence information and basic countermeasures [C ] // In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security . 2015 : 1322 - 1333 .

JANG E , GU S , POOLE B . Categorical reparameterization with gumbel-softmax [J ] . arXiv preprint arXiv:1611.01144 , 2016 .

SHOKRI R , STRONATI M , SONG C Z , et al . Membership inference attacks against machine learning models [C ] // In 2017 IEEE Symposium on Security and Privacy . 2017 : 3 - 18 .

YEOM S , GIACOMELLI I , FREDRIKSON M , et al . Privacy risk in machine learning:analyzing the connection to overfitting [C ] // In 31st IEEE Computer Security Foundations Symposium . 2018 : 268 - 282 .

SALEM A , ZHANG Y , HUMBERT M , et al . Ml-leaks:model and data independent membership inference attacks and defenses on machine learning models [C ] // In 26th Annual Network and Distributed System Security Symposium . 2019 : 24 - 27 .

LONG Y H , BINDSCHAEDLER V , GUNTER C A . Towards measuring membership privacy [J ] . CoRR,abs/1712.09136 , 2017 .

LONG Y H , BINDSCHAEDLER V , WANG L , et al . Understanding membership inferences on well-generalized learning models [J ] . CoRR,abs/1802.04889 , 2018 .

YEOM S , FREDRIKSON M , JHA S . The unintended consequences of overfitting:Training data inference attacks [J ] . CoRR,abs/1709.01604 , 2017 .

SAM D B , SURYA S , BABU R V . Switching convolutional neural network for crowd counting [C ] // 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2017 : 4031 - 4039 .

KUHA J , MILLS C . On group comparisons with logistic regression models [J ] . Sociological Methods ＆ Research , 2020 , 49 ( 2 ): 498 - 525 .

PAL M . Random forest classifier for remote sensing classification [J ] . International journal of remote sensing , 2005 , 26 ( 1 ): 217 - 222 .

SONG L , SHOKRI R , MITTAL P . Privacy risks of securing machine learning models against adversarial examples [C ] // Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security . 2019 : 241 - 257 .

SALEM A , BHATTACHARYA A , BACKES M , et al . Updates-leak:data set inference and reconstruction attacks in online learning [J ] . arXiv preprint arXiv:1904.01067 , 2019 .

HAYES J , MELIS L , DANEZIS G , et al . LOGAN:membership inference attacks against generative models [J ] . PoPETs , 2019 ( 1 ): 133 - 152 .

NASR M , SHOKRI R , HOUMANSADR A . Comprehensive privacy analysis of deep learning:passive and active white-box inference attacks against centralized and federated learning [C ] // In 2019 IEEE Symposium on Security and Privacy . 2019 : 739 - 753 .

LEINO K , FREDRIKSON M . Stolen memories:leveraging model memorization for calibrated white-box membership inference [J ] . arXiv preprint arXiv:1906.11798 , 2019 .

MELIS L , SONG C Z , CRISTOFARO E D , et al . Exploiting unintended feature leakage in collaborative learning [C ] // In 2019 IEEE Symposium on Security and Privacy . 2019 : 691 - 06 .

WANG Z B,SONG M K , Zhang Z F , Yet al . Beyond inferring class representatives:user-level privacy leakage from federated learning [C ] // In 2019 IEEE conference on Computer Communications . 2019 : 2512 - 2520 .

HITAJ B , ATENIESE G,PÉREZ-CRUZ F . Deep models under the GAN:information leakage from collaborative deep learning [C ] // In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . 2017 : 603 - 618 .

ZHU L G , LIU Z J , HAN S . Deep leakage from gradients [C ] // In Advances in Neural Information Processing Systems Annual Conference on Neural Information Processing Systems 2019 . 2019 : 14747 - 14756 .

NASR M , SHOKRI R , HOUMANSADR A . Machine learning with membership privacy using adversarial regularization [C ] // In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security . 2018 : 634 - 646 .

WANG C , LIU G Y , HUANG H J , et al . MIASec:enabling data indistinguishability against membership inference attacks in MLaaS [J ] . IEEE Transactions on Sustainable Computing , 2020 , 5 ( 3 ): 365 - 376 .

WU N , FAROKHI F , SMITH D , et al . The Value of collaboration in convex machine learning with differential privacy [J ] . IEEE Symposium on Security and Privacy , 2020 : 304 - 317 .

PATRA A , SURESH A . BLAZE:blazing fast privacy-preserving machine learning [J ] . arXiv preprint arXiv:2005.09042 , 2020 .

JIA J Y , SALEM A , BACKES M , et al . MemGuard:defending against black-box membership inference attacks via adversarial examples [C ] // In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security,CCS 2019 . 2019 : 259 - 274 .

HE Y Z , MENG G Z , CHEN K , et al . Towards privacy and security of deep learning systems:a survey [J ] . arXiv:1911.12562 , 2019 .

KESARWANI M , MUKHOTY B , ARYA V , et al . Model extraction warning in MLaaS paradigm [C ] // In Proceedings of the 34th Annual Computer Security Applications Conference,ACSAC 2018 . 2018 : 371 - 380 .

OH S J , SCHIELE B , FRITZ M . Towards reverse-engineering black-box neural networks [M ] // Explainable AI:Interpreting,Explaining and Visualizing Deep Learning . Springer,Cham , 2019 : 121 - 144 .

OREKONDY T , SCHIELE B , FRITZ M . Knockoff nets:Stealing functionality of black-box models [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 2019 : 4954 - 4963 .

CHANDRASEKARAN V , CHAUDHURI K , GIACOMELLI I , et al . Exploring connections between active learning and model extraction [J ] . arXiv preprint arXiv:1811.02054 , 2018 .

PENGCHENG L , YI J , ZHANG L . Query-efficient black-box attack by active learning [C ] // 2018 IEEE International Conference on Data Mining (ICDM) . 2018 : 1200 - 1205 .

ILYAS A , ENGSTROM L , ATHALYE A , et al . Black-box adversarial attacks with limited queries and information [J ] . arXiv preprint arXiv:1804.08598 , 2018 .

浏览量

2989

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于区块链的跨数据中心匿名可监管身份认证方案

基于人工智能的期货交易风险评估与预警方法

基于隐私集合求交的大模型双向隐私授权

基于PIR增强的可搜索加密方案研究

车-路-云协同自动驾驶安全技术研究进展