Lip forgery detection via spatial-frequency domain combination

Jiaying LIN; Wenbo ZHOU; Weiming ZHANG; Nenghai YU

doi:10.11959/j.issn.2096-109x.2022075

您当前的位置：

首页 >

文章列表页 >

Lip forgery detection via spatial-frequency domain combination

Papers and Reports | 更新时间：2024-06-03

- Lip forgery detection via spatial-frequency domain combination
- Chinese Journal of Network and Information Security Vol. 8, Issue 6, Pages: 146-155(2022)
- 作者机构：
  
  1. 中国科学院电磁空间信息重点实验室，安徽合肥 230027
  2. 中国科学技术大学网络空间安全学院，安徽合肥 230027
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2022075
  CLC： TP309.2
- Online First：2022-12，
  
  Published：15 December 2022
- 稿件说明：
移动端阅览
Jiaying LIN, Wenbo ZHOU, Weiming ZHANG, et al. Lip forgery detection via spatial-frequency domain combination[J]. Chinese Journal of Network and Information Security, 2022, 8(6): 146-155.
DOI：

Jiaying LIN, Wenbo ZHOU, Weiming ZHANG, et al. Lip forgery detection via spatial-frequency domain combination[J]. Chinese Journal of Network and Information Security, 2022, 8(6): 146-155. DOI： 10.11959/j.issn.2096-109x.2022075.

摘要

近年来，社交网络中的“换脸”视频层出不穷，对说话者进行唇型篡改是其中的视频代表之一，这给大众生活增添娱乐的同时，对于网络空间中的个人隐私、财产安全也带来了不小隐患。大多数唇型篡改检测方法在无损条件下取得了较好的表现，但广泛存在于社交媒体平台、人脸识别等场景中的压缩操作，在节约像素和时间冗余的同时，会对视频质量造成影响，破坏空域上像素与像素、帧与帧之间的连贯完整性，导致其检测性能的下降，从而引发对真实视频的错判情况。当空域信息无法提供足够有效的特征时，能够抵抗压缩干扰的频域信息就自然而然地成为重点研究对象。针对这一问题，通过分析频率信息在图像结构和梯度反馈上的优势，提出了空域频域相结合的唇型篡改检测方法，有效利用空域、频域信息的各自特点。对于空域上的唇型特征，设计了自适应提取网络和轻量级的注意力模块；对于频域上的频率特征，设计了不同分量的分离提取与融合模块。随后，通过对空域上的唇型特征和频域上的频率特征进行有侧重的融合，保留更多关键纹理信息。此外，在训练中设计细粒度约束，分开真假唇型特征类间距离的同时，拉近类内距离。实验结果表明，得益于频率信息，所提方法能有效改善压缩情况下的检测准确性，并具备一定的迁移性。另外，在对核心模块开展的消融实验中，相关结果验证了频率分量对于抗压缩的有效性，以及双重损失函数在训练中的约束作用。

Abstract

In recent years

numerous “face-swapping” videos have emerged in social networks

one of the representatives is the lip forgery with speakers.While making life more entertaining for the public

it poses a significant crisis for personal privacy and property security in cyberspace.Currently

under non-destructive conditions

most of the lip forgery detection methods achieve good performance.However

the compression operations are widely used in practice especially in social media platforms

face recognition and other scenarios.While saving pixel and time redundancy

the compression operations affect the video quality and destroy the coherent integrity of pixel-to-pixel and frame-to-frame in the spatial domain

and then the degradation of its detection performance and even misjudgment of the real video will be caused.When the information in the spatial domain cannot provide sufficiently effective features

the information in the frequency domain naturally becomes a priority research object because it can resist compression interference.Aiming at this problem

the advantages of frequency information in image structure and gradient feedback were analyzed.Then the lip forgery detection via spatial-frequency domain combination was proposed

which effectively utilized the corresponding characteristics of information in spatial and frequency domains.For lip features in the spatial domain

an adaptive extraction network and a light-weight attention module were designed.For frequency features in the frequency domain

separate extraction and fusion modules for different components were designed.Subsequently

by conducting a weighted fusion of lip features in spatial domain and frequency features in frequency domain

more texture information was preserved.In addition

fine-grained constraints were designed during the training to separate the inter-class distance of real and fake lip features while closing the intra-class distance.Experimental results show that

benefiting from the frequency information

the proposed method can enhance the detection accuracy under compression situation with certain transferability.On the other hand

in the ablation study conducted on the core modules

the results verify the effectiveness of the frequency component for anti-compression and the constraint of the dual loss function in training.

关键词

Keywords

references

SUWAJANAKORN S , SEITZ S M , KEMELMACHER-SHLIZERMAN I , . Synthesizing Obama:learning lip sync from audio [J ] . ACM Transactions on Graphics (TOG) , 2017 , 36 : 1 - 13 .

SIAROHIN A , LATHUILIÈRE S , TULYAKOV S , et al . First order motion model for image animation [J ] . ArXiv , 2019 ,abs/2003.00196.

YI R , YE Z , ZHANG J , et al . Audio-driven talking face video generation with learning-based personalized head pose [J ] . arXiv:2002.10137v2 , 2020 .

PRAJWAL K R , MUKHOPADHYAY R , NAMBOODIRI V P , et al . A lip sync expert is all you need for speech to lip generation in the wild [C ] // Proceedings of the 28th ACM International Conference on Multimedia . 2020 : 484 - 492 .

HALIASSOS A , VOUGIOUKAS K , PETRIDIS S , et al . Lips don't lie:a generalisable and robust approach to face forgery detection [C ] // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2021 : 5037 - 5047 .

FARHA Y A , GALL J . MS-TCN:multi-stage temporal convolutional network for action segmentation [C ] // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2019 : 3570 - 3579 .

QIAN Y Y , YIN G J , SHENG L , et al . Thinking in frequency:face forgery detection by mining frequency-aware clues [C ] // Proceedings of Computer Vision – ECCV 2020 . 2020 : 86 - 103 .

LI J M , XIE H T , LI J H , et al . Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection [C ] // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2021 : 6454 - 6463 .

CHEN S , YAO T P , CHEN Y , et al . Local relation learning for face forgery detection [J ] . arXiv:2105.02577 , 2021 .

REN S Q , HE K M , GIRSHICK R , et al . Faster R-CNN:towards real-time object detection with region proposal networks [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017 , 39 ( 6 ): 1137 - 1149 .

HUANG G , LIU Z , VAN DER MAATEN L , et al . Densely connected convolutional networks [C ] // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . 2017 : 2261 - 2269 .

SUN Y , WANG X G , TANG X O . Deep learning face representation from predicting 10,000 classes [C ] // Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition . 2014 : 1891 - 1898 .

WEN Y D , ZHANG K P , LI Z F , et al . A discriminative feature learning approach for deep face recognition [C ] // Proceedings of Computer Vision – ECCV 2016 . 2016 : 499 - 515 .

RÖSSLER A , COZZOLINO D , VERDOLIVA L , et al . FaceForensics++:learning to detect manipulated facial images [C ] // Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . 2019 : 1 - 11 .

LI Y Z , YANG X , SUN P , et al . Celeb-DF:a large-scale challenging dataset for DeepFake forensics [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2020 : 3204 - 3213 .

Faceswap . Faceswap github [EB ] .

THIES J , ZOLLHÖFER M , STAMMINGER M , et al . Face 2 face:real-time face capture and reenactment of RGB videos [J ] . ArXiv , 2019 ,abs/2007.14808.

DeepFakes . Deepfakes github [EB ] .

THIES J , ZOLLHÖFER M , NIEßNER M . Real-time expression transfer for facial reenactment [J ] . ACM Transactions on Graphics , 2015 , 34 ( 6 ): 1 - 14 .

LI L Z , BAO J M , ZHANG T , et al . Face X-ray for more general face forgery detection [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 2020 : 5000 - 5009 .

KINGMA D P , BA J . Adam:a method for stochastic optimization [J ] . CoRR . 2015 ,abs/1412.6980.

FRIDRICH J J , KODOVSKÝ J , . Rich models for steganalysis of digital images [J ] . IEEE Transactions on Information Forensics and Security , 2012 , 7 ( 3 ): 868 - 882 .

AFCHAR D , NOZICK V , YAMAGISHI J , et al . MesoNet:a compact facial video forgery detection network [J ] . 2018 IEEE International Workshop on Information Forensics and Security (WIFS) , 2018 : 1 - 7 .

CHOLLET F , . Xception:deep learning with depthwise separable convolutions [C ] // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . 2017 : 1800 - 1807 .

COZZOLINO D , POGGI G , VERDOLIVA L . Recasting residual-based local descriptors as convolutional neural networks:an application to image forgery detection [C ] // Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security . 2017 : 159 - 164 .

BAYAR B , STAMM M C . A deep learning approach to universal image manipulation detection using a new convolutional layer [C ] // Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security . 2016 : 5 - 10 .

RAHMOUNI N , NOZICK V , YAMAGISHI J , et al . Distinguishing computer graphics from natural images using convolution neural networks [J ] . 2017 IEEE Workshop on Information Forensics and Security (WIFS) , 2017 : 1 - 6 .

ZHOU P , HAN X T , MORARIU V I , et al . Two-stream neural networks for tampered face detection [C ] // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops . 2017 : 1831 - 1839 .

NGUYEN H H , FANG F M , YAMAGISHI J , et al . Multi-task learning for detecting and segmenting manipulated facial images and videos [C ] // Proceedings of 2019 IEEE 10th International Conference on Biometrics Theory,Applications and Systems . 2019 : 1 - 8 .

MATERN F , RIESS C , STAMMINGER M . Exploiting visual artifacts to expose deepfakesand face manipulations [J ] . 2019 IEEE Winter Applications of Computer Vision Workshops(WACVW) , 2019 : 83 - 92 .

LI Y , LYU S . Exposing DeepFake videos by detecting face warping artifacts [J ] . ArXiv , 2019 ,abs/1811.00656.

Views

286

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Phishing website detection based on two-way feature extraction and adaptive fusion

Anomaly detection in DNS: an empirical study

FL-IoV: comprehensive intrusion detection method for internet of vehicles based on federated learning

Multi-task adversarial attribution method based on hierarchical structure

Research progress of color image steganalysis

Related Author

Guo Kai

Wang Heng

Ma Ziqiang

HOU Ronghao

HE Ming

KONG Kaichuan

CHEN Yong

ZUO Peng

Related Institution

School of Information Engineering, Ningxia University

China Internet Network Information Center

School of Cyber Science and Engineering, Jinan University

Beijing Unionpay Card Technology Co., Ltd., Shenzhen Branch

School of Computer Science, Nanjing University of Posts and Telecommunications

⁰