臺灣博碩士論文加值系統

隨著深度學習技術的迅速發展，Deepfake技術已成為當今社會中引起關注的熱門話題之一。本研究的目標是比較多種不同方法和資料集在Deepfake檢測方面的性能，並評估不同資料集對檢測結果的影響。我們雇用了100位臺灣的素人演員，通過多種方法確保他們的臺灣身份。我們採用錄影棚內攝影的方式收集演員的臉孔，並取得他們的同意，生成多種Deepfake影片，以建立臺灣人自己的Deepfake 資料集，稱之為台灣高品質深偽資料庫（TaiwanHigh-qualityDeepfake dataset，簡稱 TWHD-Dataset）。我們註冊並下載了目前知名的多個深偽影像資料集，例如FaceForensics、DFDC、DeeperForensics-1.0，以及來自韓國的 KoDF 等。同時，我們選擇了一些常見的Deepfake方法，包括基於生成對抗網絡（Generative Adversarial Networks，GANs）和自編碼器（Autoencoders）等。總結來說，本研究提供了一個大型的台灣人臉深偽資料集，透過對比多種不同的Deepfake生成方法以及評估資料集對於深偽影像生成的綜合分析。這將有助於研究人員更好地理解Deepfake 生成方法的優勢和限制，並在實際應用中作出更明智的選擇。未來的研究可以進一步探索和改進Deepfake方法，以提高生成結果的真實性和可靠性。

With the rapid development of deep learning technology, Deepfake has emerged as one of the hot topics of concern in today’s society. This study aims to compare the performance of various methods and datasets in Deepfake detection and assess the impact of different datasets on detection results.

We hired 100 amateur actors from Taiwan, ensuring their Taiwanese identity through various methods. Using in-studio filming, we collected the actors’ facial images and obtained their consent to generate multiple Deepfake videos, establishing a Taiwanese-owned Deepfake dataset called the Taiwan High-quality Deepfake dataset (TWHD-Dataset).

We registered and downloaded several well-known Deepfake image datasets, such as FaceForensics, DFDC, DeeperForensics-1.0, and the Korean KoDF dataset. Additionally, we selected several commonly used Deepfake methods, including those based on Generative Adversarial Networks (GANs) and Autoencoders.

In summary, this study provides a large-scale Taiwanese facial Deepfake dataset, enabling a comprehensive analysis of different Deepfake generation methods and the evaluation of dataset impacts on Deepfake image generation. This will help researchers better understand the advantages and limitations of Deepfake generation methods and make more informed choices in practical applications. Future research can further explore and improve Deepfake methods to enhance the realism and reliability of generated results.

摘要 i
Abstract ii
目錄 iii
列表目次 v
圖目次 vi

第一章 緒論 1
1.1 深偽技術的崛起 1
1.2 豐富多樣的深度偽造技術 1
1.3 深偽技術的多種方法和高品質資料集建立 3

第二章 參考文獻 5
2.1 臉部偽造的檢測方法 5
2.2 著名的深偽資料集 5
2.2.1 Deepfake Detection Challenge (DFDC) Dataset 6
2.2.2 FaceForensics++ Dataset 8
2.2.3 Korean Deepfake Detection Dataset 9
2.2.4 Openforensics Dataset 11
2.2.5 Deeperforensics-1.0 Dataset 13
2.3 不同深偽影像資料集的比較與討論 15

第三章 研究內容與方法 20
3.1 台灣高品質深偽資料集 (TWHD-Dataset) 20
3.2 演員素材收集 20
3.3 Deepfakes 的訓練與生成 22
3.3.1 DeepFaceLab 23
3.3.2 FaceSwap 26
3.3.3 Reface: AI 28
3.4 生成 Deepfake 影片後的標註與整理 29
3.5 TWHD 資料集總結 29

第四章 實驗結果 34
4.1 主觀評估：TWHD 資料集的人眼觀察 34
4.1.1 問卷設計和受試者回應率 35
4.1.2 受試者對深偽影片的辨識能力 36
4.1.3 受試者對資料集的真假判斷 38
4.1.4 資料集的潛在缺陷與探討 40
4.2 數值評估：TWHD 資料集的客觀分析 41
4.2.1 實作細節 43
4.2.2 資料集的品質比較 44
4.2.3 資料集的深偽判別比較 47

第五章 結論與未來研究 49
5.1 研究貢獻 49
5.2 未來研究 50
5.3 研究限制 50
5.4 致謝 51

參考文獻 52

[1] O. I. Al-Sanjary, A. A. Ahmed, and G. Sulong. Development of a video tampering dataset for forensic investigation. In Forensic Science International, volume 266, pages 565–572, 2016.

[2] D. Bank, N. Koenigstein, and R. Giryes. Autoencoders. Machine learning for data science handbook: Data mining and knowledge discovery handbook, pages 353–374, 2023.

[3] Bradski, G., & Kaehler, A. Learning OpenCV: Computer vision with the OpenCV library. O’Reilly Media, Inc, 2008.

[4] L. Breiman. Random forests. Machine Learning, 45:5–32, 2001.

[5] A. Bulat and G. Tzimiropoulos. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1021–1030, 2017.

[6] C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou. Facewarehouse: A 3D facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3):413–425, 2013.

[7] F. Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1251–1258, 2017.

[8] D. Community. Deepfakes faceswap. https://github.com/deepfakes/faceswap. Accessed: June 11, 2023.

[9] C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 20:273–297, 1995.

[10] H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. K. Jain. On the detection of digital face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5781–5790, 2020.

[11] J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5203–5212, 2020.

[12] B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer. The deepfake detection challenge (DFDC) dataset. In arXiv preprint arXiv:2006.07397, 2020.

[13] B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer. The deepfake detection challenge (DFDC) preview dataset. In arXiv preprint arXiv:1910.08854, 2019.

[14] Dowson, D. C., & Landau, B. The Fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3), 450-455, 1982.

[15] C. Frith. Role of facial expressions in social interactions. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535):3453–3458, 2009.

[16] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2014.

[17] L. Guarnera, O. Giudice, and S. Battiato. Deepfake detection by analyzing convolutional traces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 666–667, 2020.

[18] Hore, A., & Ziou, D. Image quality metrics: PSNR vs. SSIM. 2010 20th International Conference on Pattern Recognition, pages 2366-2369, 2004.

[19] Y. Hu, L. Manikonda, and S. Kambhampati. What we Instagram: A first analysis of Instagram photo content and user types. In Proceedings of the International AAAI Conference on Web and Social Media, volume 8, pages 595–598, 2014.

[20] D. Huang and F. De La Torre. Facial action transfer with personalized bilinear regression. In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II, pages 144–158. Springer Berlin Heidelberg, 2012.

[21] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1501–1510, 2018.

[22] N. Hulzebosch, S. Ibrahimi, and M. Worring. Detecting CNN-generated facial images in real-world scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 642–643, 2020.

[23] V. Iglovikov and A. Shvets. Ternausnet: U-net with VGG11 encoder pre-trained on ImageNet for image segmentation. arXiv preprint arXiv:1801.05746, 2018.

[24] L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy. Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2889-2898, 2020.

[25] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.

[26] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.

[27] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

[28] P. Korshunov and S. Marcel. Deepfakes: A new threat to face recognition? Assessment and detection. In arXiv preprint arXiv:1812.08685, 2018.

[29] P. Kwon, J. You, G. Nam, S. Park, and G. Chae. KoDF: A large-scale Korean deepfake detection dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10744–10753, 2021.

[30] T. N. Le, H. H. Nguyen, J. Yamagishi, and I. Echizen. Openforensics: Large-scale challenging dataset for multi-face forgery detection and segmentation in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10117–10127, 2021.

[31] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541–551, 1989.

[32] Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu. Celeb-DF: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3207–3216, 2020.

[33] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. Advances in Neural Information Processing Systems, 30, 2017.

[34] Marek Kowalski. FaceSwap. https://github.com/MarekKowalski/FaceSwap. Accessed: June 11, 2023.

[35] F. Marra, C. Saltori, G. Boato, and L. Verdoliva. Incremental learning for the detection and classification of GAN-generated images. In 2019 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–6, 2019.

[36] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.

[37] B. Moysis, D. Christodoulou, A. Tefas, and N. Nikolaidis. Machine learning interpretability with shapley values and limited data. arXiv preprint arXiv:2002.06675, 2020.

[38] H. Nguyen, J. Yamagishi, I. Echizen, and J. F. Alegre. Fdft: A fake digital facial texture detection method based on fourier analysis. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1–9, 2017.

[39] M. Nitzan, S. Caspi, L. Wolf, and M. Irani. Face identity disentanglement via latent space mapping. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7049–7059, 2020.

[40] S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359, 2010.

[41] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, and S. F. Xu. Automatic differentiation in PyTorch. In Advances in Neural Information Processing Systems (NIPS) Workshop, 2017.

[42] C. Perez, S. Ivaldi, and M. S. Otaduy. Human perception of dynamic tactile textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4545–4553, 2020.

[43] D. Ponce-Lopez, R. Huang, S. Winkler, J. See, S. Winkler, and R. Subramanian. DAiSEE: Towards user engagement recognition in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1664–1672, 2017.

[44] M. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.

[45] M. G. Richard, M. H. Richard, and K. Y. Richard. Performance modeling of deep learning-based image super-resolution on distributed platforms. arXiv preprint arXiv:1910.07292, 2019.

[46] F. Schroff, D. Kalenichenko, and J. Philbin. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823, 2015.

[47] D. B. Shankar and J. H. Hays. Urban scene understanding from a single view based on what we expect to see. arXiv preprint arXiv:1910.07295, 2019.

[48] T. Simon, H. Joo, I. Matthews, and Y. Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1145–1153, 2017.

[49] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.

[50] L. Song, J. Zhao, and H. Xie. Attention-driven global feature and local structure learning for image emotion classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 620–621, 2020.

[51] A. Sohn, K. Aizawa, T. Ogawa, and S. L. Toward. Image inpainting using a flow-based deep generative model. In arXiv preprint arXiv:2001.04879, 2020.

[52] M. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016.

[53] M. Tan and Q. V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 6105–6114, 2019.

[54] R. Tatsumi, Y. Kanazawa, and Y. Kashima. DeepFake image detection via machine learning. arXiv preprint arXiv:1911.06878, 2019.

[55] L. Thies, P. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner. Face2Face: Real-time face capture and reenactment of RGB videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2387–2395, 2016.

[56] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4):66, 2019.

[57] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems (NIPS), pages 613–621, 2016.

[58] C. Wang, Z. Wu, and C. Liang. Robust deepfake detection via deep learning models. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI), pages 3583–3589, 2020.

[59] Y. Wu, J. Zhang, and T. Lu. A comprehensive survey on deep learning methods for deepfake detection. IEEE Access, 9:83903–83924, 2021.

[60] Z. Wu, C. Shen, and A. V. D. Hengel. Wider or deeper: Revisiting the resnet model for visual recognition. arXiv preprint arXiv:1611.10080, 2016.

[61] M. Zhang, X. Fan, and D. Zhang. Identity-invariant deepfake detection with natural face modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 666–667, 2020.

[62] X. Zhang, Y. Zhu, X. Dai, M. K. Han, C. Yuan, Y. Xie, Y. Dai, Y. Wei, and Y. Lu. Knowledge distillation from cross-task representations for action detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7036–7045, 2020.

[63] L. Zheng, W. Li, and Y. Gong. Towards open set person re-identification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7):1690–1705, 2019.

[64] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.