跳到主要內容

臺灣博碩士論文加值系統

(44.200.169.3) 您好!臺灣時間:2022/12/01 01:15
字體大小: 字級放大   字級縮小   預設字形  
回查詢結果 :::

詳目顯示

我願授權國圖
: 
twitterline
研究生:陳佳琳
研究生(外文):Chen, Chia-Lin
論文名稱:以功能性磁振造影重建具一致面部特徵之人臉影像
論文名稱(外文):Face Image Reconstruction with Consistent Facial Attributes from Functional Magnetic Resonance Imaging
指導教授:陳永昇陳永昇引用關係
指導教授(外文):Chen, Yong-Sheng
口試委員:陳麗芬彭文孝邱維辰
口試委員(外文):Chen, Li-FenPeng, Wen-HsiaoChiu, Wei-Chen
口試日期:2019-10-14
學位類別:碩士
校院名稱:國立交通大學
系所名稱:生醫工程研究所
學門:工程學門
學類:生醫工程學類
論文種類:學術論文
論文出版年:2019
畢業學年度:108
語文別:英文
論文頁數:53
中文關鍵詞:功能性磁振造影視覺影像重建重建具一致面部特徵之人臉影像深度神經網路
外文關鍵詞:Functional Magnetic Resonance ImagingVisual ReconstructionReconstruction with Consistent Facial AttributesDeep Neural Network
相關次數:
  • 被引用被引用:0
  • 點閱點閱:81
  • 評分評分:
  • 下載下載:0
  • 收藏至我的研究室書目清單書目收藏:0
功能性磁振造影(fMRI)是一種非侵入性的大腦功能造影工具,利用血氧濃度相依比的訊號推估腦部組織活化情形。當人們接受視覺刺激時,腦部細胞產生活化反應,我們期望藉此找出刺激材料與腦部反應的相關性,並且重現受試者所觀看之刺激影像
在此研究中,我們使用對應的視覺刺激與腦部反應訊號進行視覺影像重建。本論文資料收集的部分,讓受試者觀看一系列人臉影像,且在實驗中隨機出現房子的影像,受試者會進行判斷並藉此提高實驗時受試者的注意力。受試者在觀看刺激材料的同時,並收集當下的腦部反應進行下一階段的視覺重建。
在視覺重建的部分,我們主要是基於自動編碼器的架構下進行訓練,該訓練可分為三階段,第一階段使用腦部資訊與刺激材料個別訓練一個自動編碼器,得到含有原空間壓縮資訊的特徵值。第二階段是利用上一階段訓練好的模型並加入屬性解碼器一併進行訓練,使得特徵值不僅含有原空間的壓縮資訊也帶有相同的人臉屬性訊息。第三階段是訓練兩個轉換器,讓兩個不同空間的特徵值可以映射至兩空間之重疊空間,並透過兩空間的解碼器生成人臉影像與腦部反應資訊。在測試部分,我們利用在第二、三階段完成訓練的腦部資訊編碼器、轉換器與影像解碼器,使用腦部反應資訊進行人臉影像重建。
人臉影像重建結果,我們將分兩部分進行討論。第一個部分是針對40個屬性特徵預測的正確性進行討論,每一個受測者人臉重建影像在這部分的正確率都有不錯的表現(S1:81%; S2:80%; S3:81%; S4: 81%)。另一個部分是將重建結果做成線上問卷,讓人們以重建影像為判斷基礎,從兩張人臉自然影像中選出哪張與重建影像最為相像。從問卷評分結果來看,雖然答題的正確率高於chance level的題數未超過總題數的一半,但答題正確率超過七成在每個受試者資料都有十題以上。
綜上所述,讓特徵值帶有屬性資訊會使得特徵值空間限制在一定區間,故轉換器更容易將不同特性的兩特徵值轉換到共同空間。藉此我們可以使用fMRI資料重建出帶有一致特徵的人臉影像。
Functional Magnetic Resonance Imaging (fMRI) is a non-invasive measuring method for brain activity. When viewing images, visual area of subjects will be activated and their response can be recorded as fMRI data. In this work, we aim to reconstruct the image from the fMRI data evoked by visual stimuli. To be specific, we proposed a framework for face image reconstruction using the corresponding fMRI data with consistent facial attribute information.
The framework of the concept is to train a translator network which could map encoded fMRI data to the intersection feature space between fMRI data and image data. Finally, the proposed framework utilizes the translated features to reconstruct the face image with facial attribute information. To achieve the goal, we adopt three-stage in the training procedure. The first stage is the “compact-feature” stage, which extracts high-level representation from the fMRI data and stimulus images via the auto-encoder network. The second stage is a “compact-feature with attribute information” stage. We use the trained auto-encoder network for fine-tuning and add an attribute decoder in auto-encoder. Let encoded high-level representations bring not only the compressed messages but also the attribute messages. The third is a “feature transformation and cycle-reconstruction” stage. We use cycle-consistency loss, adversarial loss, reconstruction loss and feature loss to train the translator networks mapping feature to the intersection feature space and reconstruction of the face images. In the testing step, we only need the fMRI-encoder, the fMRI-translator, and image-decoder to reconstruct face image.
Using the 40 attributes, decoder result as a reconstructed image is predicted with high accuracy for every subject (S1: 0.81; S2:0.80; S3:0.81; S4: 0.81). And we evaluate the reconstruction results though an online survey. Each rater is presented with a reconstructed image and two candidate images. The rater should select the similar to the reconstructed image from candidate images. Although the average performance not more than half pass chance level. Reconstructed images of average accuracy higher than 70% have around 10 images in every subject.
In summary, the representation with consistent facial attribute information could limit the feature space. So the translator networks could easily map the two different space into the intersection space. We can reconstruct the image with consistent attributes using evoked fMRI.
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 Functional Magnetic Resonance Imaging (fMRI) . . . . . . . . . . 3
1.2.2 Autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.3 CycleGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Thesis Organization and Overview . . . . . . . . . . . . . . . . . . . . . . 5
2 Related Works 7
3 Materials and Methods 11
3.1 Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.1.1 Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.1.2 Stimuli . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.1.3 Experimental Paradigm . . . . . . . . . . . . . . . . . . . . . . . . 14
3.1.4 Data Preprocessing of fMRI data . . . . . . . . . . . . . . . . . . . 15
3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.1 Compact-Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.2 Compact-Feature with Attribute Information . . . . . . . . . . . . 21
3.2.3 Feature Transformation and cycle-Reconstruction . . . . . . . . . . 24
4 Experimental Results and Discussion 29
4.1 Result of Visual Experiment . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2 Result of Compact-Feature . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2.1 fMRI-Based Auto-encoder . . . . . . . . . . . . . . . . . . . . . . 32
4.2.2 Image-Based Auto-encoder . . . . . . . . . . . . . . . . . . . . . 33
4.3 Result of Compact-Feature with Attribute Information . . . . . . . . . . . 34
4.3.1 fMRI-Based Auto-encoder With Aattribute Information . . . . . . 34
4.3.2 Image-Based Auto-encoder with Attribute Information . . . . . . . 37
4.4 Result of Feature Transformation and cycle-Reconstruction . . . . . . . . . 38
4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.5.1 Ability of fMRIAE . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.5.2 Ability of ImgAE . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.5.3 Ability of Feature Transformation and cycle-Reconstruction . . . . 45
4.5.4 The Benefits of Feature with Attribute . . . . . . . . . . . . . . . . 45
5 Conclusions and Future Works 49
5.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Bibliography 51
[1] Simon Baron-Cohen, Howard A Ring, Sally Wheelwright, Edward T Bullmore,
Mick J Brammer, Andrew Simmons, and Steve CR Williams. Social intelligence
in the normal and autistic brain: an fmri study. European journal of neuroscience,
11(6):1891–1898, 1999.
[2] Benedikt A. Barth, Markus; Poser. Advances in high-field bold fmri. 2011.
[3] J. Deng,W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale
Hierarchical Image Database. In CVPR09, 2009.
[4] Gary H Glover. Overview of functional magnetic resonance imaging. Neurosurgery
Clinics, 22(2):133–139, 2011.
[5] Umut G¨uc¸l¨u, Ya˘gmur G¨uc¸l¨ut¨urk, Meysam Madadi, Sergio Escalera, Xavier Bar´o,
Jordi Gonz´alez, Rob van Lier, and Marcel AJ van Gerven. End-to-end semantic face
segmentation with conditional random fields as convolutional, recurrent and adversarial
networks. arXiv preprint arXiv:1703.03305, 2017.
[6] Ya˘gmur G¨uc¸l¨ut¨urk, Umut G¨uc¸l¨u, Katja Seeliger, Sander Bosch, Rob van Lier, and
Marcel AJ van Gerven. Reconstructing perceived faces from brain activations with
deep adversarial neural decoding. In Advances in Neural Information Processing
Systems, pages 4246–4257, 2017.
[7] PE Haenny and PH Schiller. State dependent activity in monkey visual cortex. i. single
cell activity in v1 and v4 on visual tasks. Experimental Brain Research, 69(2):225–
244, 1988.
[8] James V Haxby, M Ida Gobbini, Maura L Furey, Alumit Ishai, Jennifer L Schouten,
and Pietro Pietrini. Distributed and overlapping representations of faces and objects
in ventral temporal cortex. Science, 293(5539):2425–2430, 2001.
[9] Mohammad-Parsa Hosseini, Tuyen X Tran, Dario Pompili, Kost Elisevich, and
Hamid Soltanian-Zadeh. Deep learning with edge computing for localization of
epileptogenicity using multimodal rs-fmri and eeg big data. In 2017 IEEE International
Conference on Autonomic Computing (ICAC), pages 83–92. IEEE, 2017.
[10] Heng Huang, Xintao Hu, Yu Zhao, Milad Makkie, Qinglin Dong, Shijie Zhao, Lei
Guo, and Tianming Liu. Modeling task fmri data via deep convolutional autoencoder.
IEEE transactions on medical imaging, 37(7):1551–1561, 2017.
[11] Paul C Lauterbur et al. Image formation by induced local interactions: examples
employing nuclear magnetic resonance. 1973.
[12] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces
attributes (celeba) dataset. Retrieved August, 15:2018, 2018.
[13] Jeanette A Mumford, Benjamin O Turner, F Gregory Ashby, and Russell A Poldrack.
Deconvolving bold activation in event-related designs for multivoxel pattern classification
analyses. Neuroimage, 59(3):2636–2643, 2012.
[14] Kevin N Ochsner, Silvia A Bunge, James J Gross, and John DE Gabrieli. Rethinking
feelings: an fmri study of the cognitive regulation of emotion. Journal of cognitive
neuroscience, 14(8):1215–1229, 2002.
[15] Adrian M Owen, Martin R Coleman, Melanie Boly, Matthew H Davis, Steven Laureys,
and John D Pickard. Detecting awareness in the vegetative state. science,
313(5792):1402–1402, 2006.
[16] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation
learning with deep convolutional generative adversarial networks. arXiv preprint
arXiv:1511.06434, 2015.
[17] Katja Seeliger, Umut G¨uc¸l¨u, Luca Ambrogioni, Yagmur G¨uc¸l¨ut¨urk, and Marcel AJ
van Gerven. Generative adversarial networks for reconstructing natural images from
brain activity. NeuroImage, 181:775–785, 2018.
[18] Terrence J Sejnowski, Patricia S Churchland, and J Anthony Movshon. Putting big
data to good use in neuroscience. Nature neuroscience, 17(11):1440, 2014.
[19] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep
image reconstruction from human brain activity. PLoS computational biology,
15(1):e1006633, 2019.
[20] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale
image recognition. arXiv preprint arXiv:1409.1556, 2014.
[21] Heung-Il Suk, Chong-Yaw Wee, Seong-Whan Lee, and Dinggang Shen. State-space
model with deep learning for functional dynamics estimation in resting-state fmri.
NeuroImage, 129:292–307, 2016.
[22] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming
Liu. Neural encoding and decoding with deep learning for dynamic natural
vision. Cerebral Cortex, 28(12):4136–4160, 2017.
[23] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-toimage
translation using cycle-consistent adversarial networks. In Computer Vision
(ICCV), 2017 IEEE International Conference on, 2017.
電子全文 電子全文(網際網路公開日期:20241012)
連結至畢業學校之論文網頁點我開啟連結
註: 此連結為研究生畢業學校所提供,不一定有電子全文可供下載,若連結有誤,請點選上方之〝勘誤回報〞功能,我們會盡快修正,謝謝!
QRCODE
 
 
 
 
 
                                                                                                                                                                                                                                                                                                                                                                                                               
第一頁 上一頁 下一頁 最後一頁 top