基于块间光照不一致性的深度伪造检测算法

吴文轩; 周文柏; 张卫明; 俞能海

doi:10.11959/j.issn.2096-109x.2023011

您当前的位置：

首页 >

文章列表页 >

基于块间光照不一致性的深度伪造检测算法

学术论文 | 更新时间：2024-06-03

- 基于块间光照不一致性的深度伪造检测算法
- Deepfake detection method based on patch-wise lighting inconsistency
- 网络与信息安全学报 2023年9卷第1期页码：167-177
- 作者机构：
- 作者简介：
  
  [ "吴文轩（1997- ），男，浙江金华人，中国科学技术大学硕士生，主要研究方向为深度伪造检测" ]
  [ "周文柏（1992- ），男，安徽合肥人，中国科学技术大学特任副研究员，主要研究方向为信息隐藏与人工智能安全" ]
  [ "张卫明（1976- ），男，河北定州人，中国科学技术大学教授、博士生导师，主要研究方向为信息隐藏、多媒体内容安全、人工智能安全" ]
  [ "俞能海（1964- ），男，安徽无为人，中国科学技术大学教授、博士生导师，主要研究方向为多媒体信息检索、图像处理与视频通信、数字媒体内容安全" ]
- 基金信息：
  
  国家自然科学基金;The National Natural Science Foundation of China(62002334);国家自然科学基金;The National Natural Science Foundation of China(U20B2047);国家自然科学基金;The National Natural Science Foundation of China(62121002);安徽省自然科学基金;The Nature Science Foundation of Anhui Province(2008085QF296);中国科技大学探索基金项目;Exploration Fund Project of University of Science and Technology of China(YD3480002001);中央高校基础研究基金;Fundamental Research Funds for the Central Universities(WK2100000011)
- DOI：10.11959/j.issn.2096-109x.2023011
  中图分类号： TP393
- 网络首发：2023-02，
  
  纸质出版：2023-02-25
- 稿件说明：
移动端阅览
吴文轩, 周文柏, 张卫明, 等. 基于块间光照不一致性的深度伪造检测算法[J]. 网络与信息安全学报, 2023,9(1):167-177.

Wenxuan WU, Wenbo ZHOU, Weiming ZHANG, et al. Deepfake detection method based on patch-wise lighting inconsistency[J]. Chinese Journal of Network and Information Security, 2023, 9(1): 167-177.
吴文轩, 周文柏, 张卫明, 等. 基于块间光照不一致性的深度伪造检测算法[J]. 网络与信息安全学报, 2023,9(1):167-177. DOI： 10.11959/j.issn.2096-109x.2023011.

Wenxuan WU, Wenbo ZHOU, Weiming ZHANG, et al. Deepfake detection method based on patch-wise lighting inconsistency[J]. Chinese Journal of Network and Information Security, 2023, 9(1): 167-177. DOI： 10.11959/j.issn.2096-109x.2023011.

摘要

深度伪造技术的迅速发展和广泛传播引起了社会的广泛关注，但深度伪造技术的恶意应用也给社会带来了潜在威胁。因此，如何检测出此类深度伪造内容成为热门研究课题。以往的多数深度伪造检测算法着重于捕捉像素级别的细微伪造痕迹。目前的深度伪造算法大多忽略了伪造前后的光照信息，导致原始人脸与伪造人脸之间存在一定的光照不一致性，这为使用光照不一致性来检测深度伪造提供了可能。从引入光照不一致性信息和为特定任务设计网络结构模块两个角度设计了对应的算法。针对光照的引入，通过设计对应的通道融合算法，将更多的光照不一致信息提供给网络特征提取层，从而衍生出新的网络结构。为了保证该网络结构的可移植性，将特征通道融合的过程置于网络提取信息之前，从而使所提算法能够完整移植至常见的深度伪造检测网络。针对网络结构的设计，从网络结构和损失函数设计两个角度出发，提出了基于块间相似性的光照不一致性深度伪造检测算法。对于网络结构，基于伪造图像篡改区域和背景区域不一致的特性，在网络特征层中对提取特征进行分块，通过对比块间余弦相似度得到特征层相似矩阵，使网络拟合重心更偏向于光照不一致性。在此基础上，基于特征层相似性对比方案，通过将输入图像与该图像的未篡改图像进行块间真伪性对比，为这一任务设计了独立的真实数据参考及损失函数。实验结果表明，与基线算法相比，所提算法对于深度伪造检测的准确性有明显提升。

Abstract

The rapid development and widespread dissemination of deepfake techniques has caused increased concern.The malicious application of deepfake techniques also poses a potential threat to the society.Therefore

how to detect deepfake content has become a popular research topic.Most of the previous deepfake detection algorithms focused on capturing subtle forgery traces at pixel level and have achieved some results.However

most of the deepfake algorithms ignore the lighting information before and after generation

resulting in some lighting inconsistency between the original face and the forged face

which provided the possibility of using lighting inconsistency to detect deepfake.A corresponding algorithm was designed from two perspectives: introducing lighting inconsistency information and designing a network structure module for a specific task.For the introduction of lighting task

a new network structure was derived by designing the corresponding channel fusion method to provide more lighting inconsistency information to the network feature extraction layer.In order to ensure the portability of the network structure

the process of feature channel fusion was placed before the network extraction information

so that the proposed method can be fully planted to common deepfake detection networks.For the design of the network structure

a deepfake detection method was proposed for lighting inconsistency based on patch-similarity from two perspectives: network structure and loss function design.For the network structure

based on the characteristic of inconsistency between the forged image tampering region and the background region

the extracted features were chunked in the network feature layer and the feature layer similarity matrix was obtained by comparing the patch-wise cosine similarity to make the network focus more on the lighting inconsistency.On this basis

based on the feature layer similarity matching scheme

an independent ground truth and loss function was designed for this task in a targeted manner by comparing the input image with the untampered image of this image for patch-wise authenticity.It is demonstrated experimentally that the accuracy of the proposed method is significantly improved for deepfake detection compared with the baseline method.

关键词

Keywords

references

GOODFELLOW I , POUGET-ABADIE J , MIRZA M , et al . Generative adversarial networks [J ] . Communications of the ACM , 2020 , 63 ( 11 ): 139 - 144 .

HUH M , LIU A , OWENS A , et al . Fighting fake news:Image splice detection via learned self-consistency [C ] // Proceedings of Computer Vision – ECCV 2018 . 2018 : 106 - 124 .

KWON P , YOU J , NAM G , et al . KoDF:A large-scale Korean deepfake detection dataset [J ] . arXiv preprintarXiv:2103.10094 , 2021 .

LEE S . Detecting handcrafted facial image manipulations and GAN-generated facial images using shallow-fakefacenet [J ] . Applied Soft Computing , 2021 ,105:107256.

NGUYEN T T , NGUYEN C M , NGUYEN D T , et al . Deep learning for deepfakes creation and detection:a survey [J ] . arXiv:1909.11573 , 2019 .

SUN K , LIU H , YE Q X , et al . Domain general face forgery detection by learning to weight [J ] . Proceedings of the AAAI Conference on Artificial Intelligence , 2021 , 35 ( 3 ): 2638 - 2646 .

YANG X , LI Y Z , LYU S W . Exposing deep fakes using inconsistent head poses [C ] // Proceedings of ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing . 2019 : 8261 - 8265 .

ZHOU P , HAN X T , MORARIU V I , et al . Two-stream neural networks for tampered face detection [C ] // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . 2017 : 1831 - 1839 .

AFCHAR D , NOZICK V , YAMAGISHI J , et al . MesoNet:A compact facial video forgery detection network [C ] // Proceedings of 2018 IEEE International Workshop on Information Forensics and Security . 2018 : 1 - 7 .

NGUYEN H H , YAMAGISHI J , ECHIZEN I . Capsule-forensics:Using capsule networks to detect forged images and videos [C ] // Proceedings of ICASSP 2019-2019 IEEE International Conference on Acoustics,Speech and Signal Processing . 2019 : 2307 - 2311 .

LI Y Z , LYU S W . Exposing DeepFake videos by detecting face warping artifacts [J ] . arXiv preprint arXiv:1811.00656 , 2018 .

RÖSSLER A , COZZOLINO D , VERDOLIVA L , et al . FaceForensics++:manipulatedacial images [C ] // Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . 2019 : 1 - 11 .

ZHAO H , ZHOU W , CHEN D , et al . Multiattentional deepfake detection [J ] . arXiv preprint arXiv:2103.02406 , 2021 .

LIU Z Z , QI X J , TORR P H S . Global texture enhancement for fake face detection in the wild [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2020 : 8057 - 8066 .

SUN X W , WU B T , CHEN W . Identifying invariant texture violation for robust deepfake detection [J ] . arXiv preprint arXiv:2012.10580 , 2020 .

CHUGH K , GUPTA P , DHALL A , et al . Not made for each other audio visual dissonance-based deepfake detection and localization [C ] // Proceedings of MM'20:Proceedings of the 28th ACM International Conference on Multimedia . 2020 : 439 - 447 .

ZHANG X , KARAMAN S , CHANG S F . Detecting and simulating artifacts in GAN fake images [C ] // Proceedings of 2019 IEEE International Workshop on Information Forensics and Security . 2019 : 1 - 6 .

DOLHANSKY B , HOWES R , PFLAUM B , et al . The DeepFake detection challenge (DFDC) preview dataset [EB ] .

THIES J , ZOLLHÖFER M STAMMINGER M , et al . Face2Face:real-time face capture and reenactment of RGB videos [C ] // Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition . 2016 : 2387 - 2395 .

DeepFakes [EB ] .

FaceSwap [EB ] .

LI L Z , BAO J M , YANG H , et al . FaceShifter:Towards high fidelity and occlusion aware face swapping [EB ] .

LI L Z , BAO J M , YANG H , et al . FaceShifter:Towards high fidelity and occlusion aware face swapping [J ] . arXiv preprint arXiv:1912.13457 , 2019 .

KOUJAN M R , DOUKAS M C , ROUSSOS A , et al . Head2Head:Video-based neural head synthesis [C ] // Proceedings of 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition . 2020 : 16 - 23 .

ZHU H , FU C , WU Q , et al . AOT:appearance optimal transport based identity swapping for forgery detection [C ] // NeurIPS,IEEE , 2020 .

ZHU H , FU C Y , WU Q Y , et al . AOT:appearance optimal transport based identity swapping for forgery detection [C ] // Proceedings of NeurIPS . 2020 .

MASI I , KILLEKAR A , MASCARENHAS R M , et al . Two branch recurrent network for isolating deepfakes in videos [C ] // Proceedings of Computer Vision – ECCV 2020 , 2020 : 667 - 684 .

WU X , XIE Z , GAO Y T , et al . SSTNet:detecting manipulated faces through spatial,steganalysis and temporal features [C ] // Proceedings of ICASSP 2020 - 2020 IEEE International Conference on Acoustics,Speech and Signal Processing . 2020 : 2952 - 2956 .

SHRUTI AL , HANY F , YUMING G , et al . Protecting world leaders against deep fakes [C ] // Proceedings of CVPR Workshops . 2019 .

LI L Z , BAO J M , ZHANG T , et al . Face X-ray for more general face forgery detection [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2020 : 5000 - 5009 .

LIU H G , LI X D , ZHOU W B , et al . Spatial-phase shallow learning:Rethinking face forgery detection in frequency domain [C ] // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2021 : 772 - 781 .

BRAINARD D H , WANDELL B A . Analysis of the retinex theory of color vision [J ] . Journal of the Optical Society of America A,Optics and Image Science , 1986 , 3 ( 10 ): 1651 - 1661 .

BUCHSBAUM G . A spatial processor model for object colour perception [J ] . Journal of the Franklin Institute , 1980 , 310 ( 1 ): 1 - 26 .

VAN DE WEIJER J , GEVERS T , GIJSENIJ A . Edge-based color constancy [J ] . IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society , 2007 , 16 ( 9 ): 2207 - 2214 .

FINLAYSON G D , TREZZI E . Shades of gray and colour constancy [C ] // Proceedings of Color and Imaging Conference . 2004 .

JOZE H R V , DREW M S , FINLAYSON G D , et al . The role of bright pixels in illumination estimation [C ] // Proceedings of Color and Imaging Conference . 2012 .

YANG K F , GAO S B , LI Y J . Efficient illuminant estimation for color constancy using grey pixels [C ] // Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition . 2015 : 2254 - 2263 .

QIAN Y L , KÄMÄRÄINEN J K , NIKKANEN J , et al . On finding gray pixels [C ] // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2019 : 8054 - 8062 .

GATTA C , RIZZI A , MARINI D . ACE:an automatic color equalization algorithm [C ] // Proceedings of Conference on Colour in Graphics . 2002 .

LAND E H , MCCANN J J . Lightness and retinex theory [J ] . Journal of the Optical Society of America , 1971 , 61 ( 1 ): 1 - 11 .

FRIDRICH J , KODOVSKY J . Rich models for steganalysis of digital images [J ] . IEEE Transactions on Information Forensics and Security , 2012 , 7 ( 3 ): 868 - 882 .

浏览量

173

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于孪生网络的鲁棒性深度伪造检测方法