臺灣博碩士論文加值系統

DeepFake 影片技術近幾年對社會帶來許多危害，為了減少 DeepFake 對社群媒體帶來的負面影響，許多 DeepFake 檢測方法已被提出，並在公開資料集上表現出高準確度的結果。然而，這些高準確率的方法，都依賴良好的人臉檢測結果或是過於複雜的特徵。不好的人臉檢測結果或是過於複雜的特徵，可能會導致 DeepFake 檢測技術面臨新的挑戰，不好的人臉偵測結果可能會影響檢測器的性能，過多的特徵增加訓練時間，甚至可能影響檢測器的性能。
為了解決上述問題，本文提出了一種新的強健 DeepFake 影片檢測方法，即帶有遮蔽數據增強的特徵糾纏感知網路（FEANMA）。該方法首先使用傳統卷積神經網路提取空間特徵，然後利用時空特徵萃取器，探索不同幀和像素之間的關聯性。最後，引入特徵糾纏學習（FEL），把時空特徵融合在一起後餵入 Transformer，來進一步提高模型的穩健性，並減少特徵複雜度。
FEANMA 部分實驗，在訓練階段會隨機把部分幀的人臉圖像替換成背景，再利用生成器，將無效人臉圖像重建為有效的人臉圖像。因此，即使人臉檢測結果不穩定，FEANMA 中的時空特徵仍然可以維持檢測性能。實驗結果表示，所提出的 FEANMA 實現了最先進的性能。

DeepFake video technology has brought many dangers to society in recent years. To reduce the impact of DeepFake on social media, many DeepFake detection methods have been proposed and demonstrated excellent detection performance on public datasets. However, these high-performance methods heavily rely on good face detection results or complex features. Poor face detection results or complex features may encounter new challenges on DeepFake detection, as poor face detection may affect the performance and too many features may increase training time and even affect the detector's performance.
To address these issues, this paper proposes a new robust DeepFake video detection method, called the Feature Entanglement-Aware Network (FEAN). The method first uses a traditional convolutional neural network to extract spatial features and then uses a spatiotemporal feature extractor to explore the correlation between different frames and pixels. Finally, Feature Entanglement Learning (FEL) is introduced to fuse the spatiotemporal features together and fed into a transformer to further improve the model's robustness and reduce feature complexity.
In the FEAN experiments, during the training phase, some frames are randomly replaced, and the invalid face images are reconstructed into valid face images using a generator. Therefore, even if the face detector is unstable, the spatiotemporal features in FEAN can still maintain good performance. Overall, the experiments show that the proposed FEAN achieves state-of-the-art performance.

中文摘要 I
Abstract II
誌謝 XI
目錄 XII
表目錄 XIV
圖目錄 XVI
第一章 緒論 1
1.1. 研究動機 1
1.2. 本文結構 4
1.3. 貢獻 5
第二章 相關研究 6
2.1. 基於空間特徵的深偽影像檢測 6
2.2. 基於時間和空間特徵的深偽影像檢測 7
2.3. 對抗式攻擊 8
第三章 研究方法 9
3.1. 概述 9
3.2. 時空特徴萃取 10
3.3. 特徵糾纏學習 13
3.4. 遮蔽數據增強 15
3.5. 損失函數 16
第四章 實驗結果 18
4.1. 資料集 18
4.2. 實驗設置 18
4.2.1 資料前處理 18
4.2.2 訓練設置 19
4.2.3 評分方式 20
4.3. 基於標準數據增強的定量結果 20
4.4. 基於隨機遮蔽的定量結果 22
4.5. 消融研究 26
4.5.1 訓練次數 26
4.5.2 數據在訓練和測試期間的影片使用幀數 30
4.5.3 數據在訓練和測試期間的 Transformer Block 數量 31
4.5.4 數據在訓練和測試期間的 Multi-head 數量 32
4.5.5 數據在訓練期間的遮蔽比例 32
4.5.6 損失函數權重 33
4.5.7 不同維度之特徵糾纏比較 34
第五章 結論 39
參考文獻 40

[1] Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: a compact facial video forgery detection network. In 2018 IEEE international workshop on information forensics and security (WIFS), pages 1-7. IEEE, 2018.
[2] Irene Amerini, Leonardo Galteri, Roberto Caldelli, and Alberto Del Bimbo. Deepfake video detection through optical flow based cnn. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0-0, 2019.
[3] Shumeet Baluja and Ian Fischer. Adversarial transformation networks: Learning to generate adversarial examples. arXiv preprint arXiv: 1703.09387,2017.
[4] Belhassen Bayar and Matthew C Stamm. A deep learning approach to universal image manipulation detection using a new convolutional layer. In Proceedings of the 4th ACM workshop on information hiding and multimedia security, pages 5-10, 2016.
[5] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020.
[6] Avishek Joey Bose and Parham Aarabi. Adversarial attacks on face detectors using neural net based constrained optimization. In 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), pages 1-6. IEEE, 2018.
[7] Zhixi Cai, Kalin Stefanov, Abhinav Dhall, and Munawar Hayat. Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization. arXiv preprint arXiv:2204.06228, 2022.
[8] Nicholas Carlini and Hany Farid. Evading deepfake-image detectors with white and black-box attacks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 658-659, 2020.
[9] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39-57. Ieee, 2017.
[10] Francois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[11] Umur Aybars Ciftci, Ilke Demir, and Lijun Yin. Fakecatcher: Detection of synthetic portrait videos using biological signals. IEEE transactions on pattern analysis and machine intelligence, 2020.
[12] Davide Alessandro Coccomini, Nicola Messina, Claudio Gennaro, and Fabrizio Falchi. Combining efficientnet and vision transformers for video deepfake detection. In Image Analysis and Processing-ICIAP 2022: 21st International Conference, Lecce, Italy, May 23-27, 2022, Proceedings, Part III, pages 219-229. Springer, 2022.
[13] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 5781-5790, 2020.
[14] Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. The deepfake detection challenge (dfdc) dataset. arXiv preprint arXiv:2006.07397,2020.
[15] Joel Frank, Thorsten Eisenhofer, Lea Schonherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In International conference on machine learning, pages 3247-3258. PMLR, 2020.
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11) :139-144, 2020.
[17] lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1412.6572, 2014.
[18] Zhihao Gu, Yang Chen, Taiping Yao, Shouhong Ding, Jilin Li, and Lizhuang Ma. Delving into the local: Dynamic inconsistency learning for deepfake video detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 744-752,2022.
[19] Zhihao Gu, Taiping Yao, Yang Chen, Shouhong Ding, and Lizhuang Ma. Hierarchical contrastive inconsistency learning for deepfake video detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII, pages 596-613. Springer, 2022.
[20] Zhihao Gu, Taiping Yao, C Yang, Ran Yi, Shouhong Ding, and Lizhuang Ma. Region-aware temporal inconsistency learning for deepfake video detection. In Proceedings of the 31th International Joint Conference on Artificial Intelligence, volume 1, 2022.
[21] David Guera and Edward J Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS), pages 1-6. IEEE, 2018.
[22] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't lie: A generalisable and robust approach to face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5039-5049, 2021.
[23] Chih-Chung Hsu, Chia-Yen Lee, and Yi-Xiu Zhuang. Learning to detect fake face images in the wild. In 2018 international symposium on computer, consumer and control (IS3C), pages 388-391. IEEE, 2018.
[24] Chih-Chung Hsu, Yi-Xiu Zhuang, and Chia-Yen Lee. Deep fake image detection based on pairwise learning. Applied Sciences, 10(1):370, 2020.
[25] Juan Hu, Xin Liao, Jinwen Liang, Wenbo Zhou, and Zheng Qin. Finfer: Frame inference-based deepfake detection for high-visual-quality videos. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 951-959, 2022.
[26] Shehzeen Hussain, Paarth Neekhara, Malhar Jere, Farinaz Koushanfar, and Julian McAuley. Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3348-3357, 2021.
[27] Davis E King. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research, 10:1755-1758, 2009.
[28] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[29] Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 12(4):307-392, 2019.
[30] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5001-5010, 2020.
[31] Yuezun Li, Ming-Ching Chang, and Siwei Lyu. In ictu oculi: Exposing ai created fake videos by detecting eye blinking. In 2018 IEEE international workshop on information forensics and security (WIFS), pages 1-7. IEEE, 2018.
[32] Yuezun Li and Siwei Lyu. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv: 1811.00656, 2018.
[33] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3207-3216, 2020.
[34] lacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt, and Wael AbdAlmageed. Two-branch recurrent network for isolating deepfakes in videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pages 667-684. Springer, 2020.
[35] Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. Emotions don't lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the 28th ACM international conference on multimedia, pages 2823-2832,2020.
[36] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
[37] Paarth Neekhara, Brian Dolhansky, Joanna Bitton, and Cristian Canton Ferrer. Adversarial threats to deepfake detection: A practical perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 923-932, 2021.
[38] Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency-aware clues. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII, pages 86-103. Springer, 2020.
[39] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias NieBner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1-11,2019.
[40] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan. Recurrent convolutional strategies for face manipulation detection in videos. Interfaces (GUI), 3(1):80-87, 2019.
[41] Yu Yang Sun, Zhi Yong Zhang, Isao Echizen, Huy H Nguyen, ChangZhen Qiu, and Lu Sun. Face forgery detection based on facial region displacement trajectory series. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 633-642.2023.
[42] Zekun Sun, Yujie Han, Zeyu Hua, Na Ruan, and Weijia Jia. Improving the efficiency and robustness of deepfakes detection through precise geometric features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3609-3618,2021.
[43] Mingxing Tan and Quoc Le. Eficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR. 2019.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. stat, 1050(20):10-48550, 2017.
[46] Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14923-14932, 2021.
[47] Ping Wang, Kunlin Liu, Wenbo Zhou, Hang Zhou, Honggu Liu, Weiming Zhang, and Nenghai Yu. Adt: Anti-deepfake transformer. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2899-1903. IEEE, 2022.
[48] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8695-8704, 2020.
[49] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pages 6861-6871. PMLR, 2019.
[50] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24, 2020.
[51] Jia Xiang and Gengming Zhu. Joint face detection and facial expression recognition with mtcnn. In 2017 4th international conference on information science and control engineering (ICISCE), pages 424 427. IEEE, 2017.
[52] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of the IEEE international conference on computer vision, pages 1369-1378, 2017.
[53] Daichi Zhang, Fanzhao Lin, Yingying Hua, Pengju Wang, Dan Zeng, and Shiming Ge. Deepfake video detection with spatiotemporal dropout transformer. arXiv preprint arXiv: 2207.06612,2022.
[54] Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: a comprehensive review. Computational Social Networks, 6(1):1-23, 2019.
[55] Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Neng-hai Yu. Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2185-2194, 2021.
[56] Yi-Xiu Zhuang and Chih-Chung Hsu. Detecting generated image based on a coupled network with two-step pairwise learning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 3212-3216. IEEE, 2019.
[57] Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. Wilddeepfake: A challenging real-world dataset for deepfake detection. In Proceedings of the 28th ACM international conference on multimedia, pages 2382-2390, 2020.