臺灣博碩士論文加值系統

傳統相機在拍攝過後，成像的景深以及焦點都無法再改變。但隨著光場技術的發明與進步，人們現在可以透過拍攝多張不同視角的照片後，藉由電腦運算重新調整對焦深度與光圈大小，這項被稱為數位重對焦的技術在近幾年得到科技公司的重視。舉例來說現在的智慧型手機上已經開始配置更多的鏡頭，來提供消費者更強大的攝像功能。多鏡頭手機其中一個重要的功能，就是能讓使用者用拍出單眼相機大光圈的景深效果，數位重對焦就是其中一種實現淺景深效果的方式。然而現在多鏡頭手機上的淺景深功能並非數位重對焦的結果，它無法自由對焦在所有深度，只能將前景背景分離後，針對背景做虛化模糊。過去黃朝宗等人曾提出快速的對焦演算法，它能提供物理擬真的數位重對焦影像。然而有關於這個演算法的硬體架構設計並沒有被討論過。在這篇論文中我們將會針對該演算法中計算最繁重的視點內插，探討如何將其實現到數位晶片之中。

在數位重對焦的演算法中，我們必須使用高度平行化的視點內插引擎。然而在以往的做法中並沒有考慮多組視點內插引擎平行時會遇到的問題。例如龐大的電路內部記憶體消耗，以及吞吐量無法隨著硬體數量線性上升等等。為了解決此問題，本篇論文探討了不同層面的平行方式，並且提出了行平行與視點平行的硬體設計方法來節省記憶體使用量以及突破吞吐量的限制。透過行平行的架構我們節省了70\%的記憶體面積並且增加了8\%的吞吐量。接著再透過視點平行的架構能將整體的吞吐量最高提高至7.5G pixel/second。

在這篇論文的最後我們提供了兩個設計範例：單視點合成引擎以及多視點合成引擎。我們使用台積電四零奈米半導體製程進行合成，單視點合成引擎使用了300萬個邏輯閘以及2.25KB的晶片內記憶體。多視點合成引擎則使用了1600萬個邏輯閘以及3KB的晶片內記憶體。當運作在200MHz的頻率下時，他們分別可以提供2.0G以及6.7G pixel/sec的吞吐量。而功耗的部份我們透過動態功耗分析工具(PrimeTime)在TT-corner進行估計，結果分別是32.5mW以及75.7mW。

Conventional cameras cannot alter the aperture size and focus depth after the pictures were taken. With the invention and progress made in the light field technology, people now can change the aperture size and focus depth on computers afterwards. This technique called digital refocusing is now attracting much attention than ever before. For instance, smartphones nowadays are putting more cameras to enhance the picture quality and to provide more fancy effects. Simulating the shallow depth of field effect created by conventional cameras with big aperture is one of them. However, current solutions are not digital refocusing algorithm. They create this shallow depth of field effect by blurring the background region. Huang et al. proposed a fast digital refocusing algorithm which can provide photorealistic results. However the VLSI architecture of this algorithm hasn't be discussed. In this thesis, we focus on implementing the most time-consuming part of this algorithm---view synthesis---into ASIC.

The refocusing algorithm need a highly paralleled view synthesis engine to render a great amount of pixels. Li-Ren Huang designed a VLSI architecture which can compute multiple pixels in parallel. Nevertheless, he did not consider the potential problems for further parallelism, e.g. multiple engines working simultaneously. The pixel throughput cannot grow linearly with the increase of hardware resources. And the huge on-chip memory usage is also a problem. In this thesis, we discuss different levels of view synthesis parallelism, and we propose two new VLSI design methodologies: line-level and view-level parallelism. With line-level parallelism, we save 70\% on-chip memories and increase 8\% throughput of rendered pixels. We further boost the throughput up to 7.5G pixel/sec after incorporating the view-level parallelism architecture.

Two design examples called single-view renderer and multi-view renderer are given in this thesis. They are both synthesized under TSMC 40nm technology process. Single-view renderer cost 0.3M gates and 2 kbytes on-chip memories. Multi-View renderer cost 1.6M gates and 3 kbytes on-chip memories. They can provide 2.0G and 6.7G pixel/sec throughput when operating at 200MHz. The power is measured using dynamic power analysis tool at TT-corner, 200MHz. And the results are 32.5mW and 75.7mW respectively.

1 Introduction 1
1.1 Motivation 1
1.2 Related Work 3
1.2.1 Refocusing Techniques 3
1.2.2 1D Line-Scan View Synthesis 6
1.2.3 Block-Based Multi-Rate Refocusing for Sparse Light Fields 9
1.3 Thesis Organization 10
2 Architecture Design of Multi-Line/Multi-View Renderer 11
2.1 Level of Parallelism for View Synthesis in Refocusing 11
2.1.1 Pixel-Level Parallelism 12
2.1.2 Line-Level Parallelism 12
2.1.3 View-Level Parallelism 12
2.1.4 Quick Summary 14
2.2 Design Methodology for Multi-Line Renderer 14
2.2.1 Design Constraints of Multi-Line Renderer 18
2.2.2 Direct Architecture Using Line Buffers 19
2.2.3 Proposed Architecture Using Memory Sharing 22
2.2.4 Summary of Multi-Line Renderer 32
2.3 Design Methodology for Multi-View Renderer 35
2.3.1 Multi-View Architecture Using Pixel Splatting 36
2.3.2 Summary of Multi-View Renderer 40
3 Implementation of Multi-View/Multi-Line Renderer for Real-Time Block-Based Refocusing 43
3.1 Target System Overview 43
3.2 Implementation Results 45
4 Conclusion and Future Work 51
4.1 Conclusion 51
4.2 Future Work 52

[1] C. Huang, Y. Wang, L. Huang, J. Chin, and L. Chen, “Fast physically correct refocusing for sparse light fields using block-based multi-rate view interpolation,” IEEE Transactions on Image Processing, vol. 26, no. 2, pp. 603–618, Feb 2017.
[2] Shu-Wen Yang, “VLSI architecture and system design of fast realistic block- based refocusing for Full-HD sparse light fields,” 2018.
[3] Li-Ren Huang, “Analysis and VLSI architecture design of 1D view synthesis algorithm for fast realistic refocusing,” 2016.
[4] P. Rokita, “Generating depth of-field effects in virtual reality applications,” IEEE Computer Graphics and Applications, vol. 16, no. 2, pp. 18–21, March 1996.
[5] Yuichi Taguchi, Amit Agrawal, Ashok Veeraraghavan, Srikumar Rama- lingam, and Ramesh Raskar, “Axial-cones: Modeling spherical catadioptric cameras for wide-angle light field rendering,” in ACM SIGGRAPH Asia 2010 Papers, New York, NY, USA, 2010, SIGGRAPH ASIA ’10, pp. 172:1–172:8, ACM.
[6] Francesc Moreno-Noguer, Peter N. Belhumeur, and Shree K. Nayar, “Active refocusing of images and videos,” in ACM SIGGRAPH 2007 Papers, New York, NY, USA, 2007, SIGGRAPH ’07, ACM.
[7] W. Zhang and W. Cham, “Single-image refocusing and defocusing,” IEEE Transactions on Image Processing, vol. 21, no. 2, pp. 873–882, Feb 2012. 54
[8] Ren Ng, “Fourier slice photography,” in ACM SIGGRAPH 2005 Papers, New York, NY, USA, 2005, SIGGRAPH ’05, pp. 735–744, ACM.
[9] Todor Georgeiv, Ke Colin Zheng, Brian Curless, David Salesin, Shree Na- yar, and Chintan Intwala, “Spatio-angular resolution tradeoffs in integral photography,” in Proceedings of the 17th Eurographics Conference on Ren- dering Techniques, Aire-la-Ville, Switzerland, Switzerland, 2006, EGSR ’06, pp. 263–272, Eurographics Association.
[10] Chia-Kai Liang, Tai-Hsu Lin, Bing-Yi Wong, Chi Liu, and Homer H. Chen, “Programmable aperture photography: Multiplexed light field acquisition,” in ACM SIGGRAPH 2008 Papers, New York, NY, USA, 2008, SIGGRAPH ’08, pp. 55:1–55:10, ACM.
[11] Benjamin Huhle, Timo Schairer, Philipp Jenke, and Wolfgang Straber, “Re- alistic depth blur for images with range data,” in Proceedings of the DAGM 2009 Workshop on Dynamic 3D Imaging, Berlin, Heidelberg, 2009, Dyn3D ’09, pp. 84–95, Springer-Verlag.
[12] Test Model 8 of 3D-HEVC and MV-HEVC, ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 document JCT3V-H1003, Apr. 2014.
[13] Y. Horng, Y. Tseng, and T. Chang, “VLSI architecture for real-time HD1080p view synthesis engine,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 21, no. 9, pp. 1329–1340, Sept 2011.
[14] P. Tsung, P. Lin, K. Chen, T. Chuang, H. Yang, S. Chien, L. Ding, W. Chen, C. Cheng, T. Chen, and L. Chen, “A 216fps 4096×2160p 3DTV set-top box SoC for free-viewpoint 3DTV applications,” in 2011 IEEE International Solid-State Circuits Conference, Feb 2011, pp. 124–126.
[15] M. Schaffner, P. Greisen, S. Heinzle, F. K. Gürkaynak, H. Kaeslin, and A. Smolic, “MADmax: A 1080p stereo-to-multiview rendering ASIC in 65 nm CMOS based on image domain warping,” in 2013 Proceedings of the ESSCIRC (ESSCIRC), Sept 2013, pp. 61–64.