臺灣博碩士論文加值系統

近年來，隨著科技的快速發展以及對於拍照的需求，數位相機已經成為人們生活中不可或缺的一部分。一般來說，數位相機是經由鏡頭擷取影像資料，並再經由一連串的處理程序後，送到壓縮處理單元並且儲存在記憶體中。而為了降低成本與體積，目前數位相機都是採 用單一影像感測器，也就是在影像感測器前面放上彩色濾波片，使得每一像素只有得到紅、綠、藍其中一種顏色，再藉由彩色濾波片內插演算法透過周圍像素的值來預測而其他兩種顏色。
一般來說，經由此內插後而得到的圖常會出現兩種缺陷，分別是拉鍊效應(zipper effect)以及不真實顏色(false color)。許多著名的內插演算法都是要去解決這兩種缺陷來達到較好的影像品質。然而，在這些之中，並沒有任何一個演算法有針對硬體的考量來設計。因此，在本篇論文中，我們分析了這些演算法的硬體成本並且根據分析所得到的結果，提出了以硬體設計為導向的彩色濾波片內插演算法。這個方法是採用了以彩度變化量為權重的內插方式。實驗結果顯示在所提出的方法不但能達到低成本硬體的考量，並且仍然可以達到好的影像品質。這也顯示了所提出的方法對於影像品質與硬體成本之間是很好的平衡點。
此外，我們也針對一般數位相機影像處理的解法進行評估，包括數位訊號處理器(DSP)、特定應用積體電路(ASIC)、混合式的架構以及粗顆粒可重組化影像處理器(CRISP).數位訊號處理器擁有高彈性可支援多樣的影像處理演算法，然而並不能達到預覽模式下即時處理的需求。而ASIC 則是對於特定的演算法針對面積、功率以及速度進行最佳化，因此擁有高效能以及低成本的優勢，但是並不具備架構的彈性，無法達到對於照相時所需不同高畫質影像處理管線演算法的特性。而混合式的架構則是結合了上述兩種方式的優點，然而硬體成本過高而且使用效率也相當低。因此，粗顆粒可重組化影像處理器(CRISP)則是被提出來解決傳統影像處理器所遇到的問題。根據影像處理演算法之高度運算相似性及硬體資源共享性，CRISP是以預覽模式所需要的影像處理管線為硬體投資的標準，使用足夠的硬體資源來滿足及時化的需求並且投資更多適當的硬體和繞線來增加架構的彈性和處理能力。如此除了兼顧ASIC和DSP 的優點外，CRISP 還是一個擁有低成本及高效率的影像處理器。
因此在本篇論文中，為了提升在預覽模式下所得到的影像與視訊的品質，我們採用了CRISP 的設計概念並且提出了雙資料流影像處理器(CRISP-DS)，其全新的設計概念稱作Dual-Stream with Context Switch。這概念則是結合了CRISP 在預覽模式及拍照模式下的特性。在預覽模式下，透過CRISP-DS 連結影像感測器介面中的鎖相迴路(PLL)來產生比影像感測器快一倍的頻率，並且與原本影像感測器的頻率進行同步的動作。然後輸入的資料流與同步訊號在此介面中則被修改成每兩個時脈(clock cycle)來處理一筆資料，也就是說對於每個的處理單元來說都有一個時脈是沒有動作的。因此，藉由增加每個處理單元的context 使得新增加的context 可以在這個時脈中對處理單元進行重組。如此每個處理單元就有如變成兩倍一樣，而所能支援的影像管線處理演算法的能力也就因此大大提升。也就是說在此設計概念下，對於同樣一套較好的影像處理管線演算法，相較於CRISP 必須透過增加處理單元來達成，CRISP-DS 只需要增加一組新的context 就可以達到相同的效果。如此就可以大幅度的降低硬體成本並且維持低成本的優勢。
在硬體實作方面，CRISP-DS 已經實作成晶片，採用TSMC 0.13 um 1P8M 的製程。晶片的面積為3.18mm x 3.18mm 而晶片核心大小為 2.2mm x 2.2mm。最高的頻率可以達到200Mhz，而在預覽模式下則是跑在比影像感測器的頻率還要快兩倍的頻率。當跑在200Mhz時，平均消耗的功率為314mW。在CRISP-DS 中我們總共使用了104,192bits 的SRAM。在比較上，透過所提出的測試影像處理管線演算法，實作結果證明CRISP-DS 在面積與使用功率的效率上都是比CRISP 還來的優異，處理速度也快上六倍。而與TI 的TMS320C64x相比，處理速度則是快了250 倍以上。

In recent years, the Digital Still Camera (DSC) has been widely adopted as an image capture device for PC-based multi-media system. A digital still camera can capture images by using a CCD or CMOS sensor and then compress the data to store the images on a memory card. In order to reduce the cost and size, many digital cameras use a single image sensor with a color filter array (CFA) to capture images. Because each pixel only contains one of the three primary colors in CFA, the others must be estimated from the neighboring pixels. This process is called as CFA demosaicking.
Generally speaking, there are two artifacts generated in the process of interpolation,including zipper effect and false color artifacts. Many state-of-the-art demosaicking methods are proposed to reduce these artifacts for achieving better image quality perceptually and in PSNR. However, hardware cost for VLSI implementation is not considered in most of them. Therefore, a hardware-oriented CFA interpolation algorithm is developed in this thesis according to the hardware cost analysis results. The proposed algorithm is by use of chrominance variance weighting scheme interpolation. Experimental results show that our method can achieve good image quality in PSNR than the existing methods on variety of test images while low hardware cost is still maintained. It shows that this method can be a good compromise between image quality and hardware cost.
Besides, we also evaluate mostly used existing solutions to image processor, including DSP, ASIC, hybrid solution and CRISP [2]. DSP owns high flexibility and can handle almost all kinds of image pipeline tasks in a DSC system, but the cost is much higher and it can’t meet the real-time requirement in the preview mode. The application-specific-integrated-circuit (ASIC) solution is highly optimized in terms of area, power and speed to perform its designated task, but it, however, doesn’t have much flexibility for different algorithms in the picture-taking mode. The hybrid solution combines the advantages of DSP and ASIC but the hardware utilization is low and the cost is too high. Therefore, a coarse-grain reconfigurable image processor, CRISP, has been proposed to solve this problem. It can approach the hardware cost lower bound of preview engine caused by the real-time constraints in the preview mode. In addition, the flexibility requirement in the picture-taking mode can be achieved by the reconfigurability of CRISP. The high processing speed can also be achieved by the processing elements specially designed for image processing tasks by utilizing the algorithmic similarity. In summary, CRISP combines the advantages of ASIC and DSP into a single hardware by proper time-space tradeoff in different modes. The high flexibility and efficiency of CRISP is very suitable for image pipeline in DSC.
Therefore, in order to upgrade the image or video quality in the preview mode, we follow the concept of CRISP and propose a dual-stream reconfigurable image stream processor (CRISP-DS) with a new design idea, called dual stream with context switch. The concept of dual-stream with context switch is to combining these characteristics in the preview and picture-taking mode. In the preview mode, inside the interface of CRISP-DS, there is a PLL (Phase lock loop) to generate the frequency twice than sensor frequency and to synchronize it. Then the data stream and synchronization signal are modified such that there are two cycle to process a pixel, which concept is similar to data stream rate adjustment in the picture-taking mode. Then we double the context for some reconfigurable stage processing elements (RSPEs) and each context is corresponding to the each one of two cycles, which means each RSPE can be configured as two different image pipeline algorithms. Then a better image pipeline can be employed in the concept meanwhile the overhead of gate count to implement this pipeline is mush smaller than original CRISP design.
The CRISP-DS chip is fabricated with TSMC 0.13um 1P8M CMOS process via CIC. The chip die size is 3.18 mm x 3.18 mm and the core size is 2.2 mm x 2.2 mm. The max working frequency in preview mode is twice than the frequency of the sensor and 200Mhz in picture-taking mode. The average total power consumption at 200 MHz is 314 mW. The total on-chip SRAM bit number is 104,192 bits. For the golden test image pipeline, the implementation results demonstrate that the area and power efficiency of CRISP-DS are better than CRIPS. In additions, the processing speed of CRISP-DS is 6 times faster than CRISP and over 250 times fasters than TMS320C64x DSP.

Abstract xi
1 Introduction 1
1.1 Image Pipeline Overview . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Importance of Color Filter Array Interpolation . . . . . . . . . . . . . . 2
1.3 Existing solution to Image Processor . . . . . . . . . . . . . . . . . . . 4
1.3.1 Digital Signal Processor (DSP) . . . . . . . . . . . . . . . . . . 4
1.3.2 Application Specific Integrated Circuit (ASIC) . . . . . . . . . 4
1.3.3 Hybrid Architecture of DSP and ASIC . . . . . . . . . . . . . . 5
1.3.4 CRISP: Coarse-grained Reconfigurable Image Signal Processor 6
1.4 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Introduction to CFA Interpolation Algorithms 9
2.1 Non-Adaptive Algorithms . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.1 Bilinear Interpolation . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.2 Smooth Hue Transition (SHT) . . . . . . . . . . . . . . . . . . 10
2.1.3 ECI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Adaptive Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Edge Sensing Interpolation . . . . . . . . . . . . . . . . . . . . 12
2.2.2 Laplacian Interpolation . . . . . . . . . . . . . . . . . . . . . . 14
2.2.3 Lu’s weighting method . . . . . . . . . . . . . . . . . . . . . . 15
2.2.4 PCSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Iteration Scheme Algorithms . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.1 Alternating Projections (AP) . . . . . . . . . . . . . . . . . . . 18
2.3.2 Successive Approximation (SA) . . . . . . . . . . . . . . . . . 20
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3 Algorithm Development 23
3.1 VLSI Hardware Cost Analysis . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Proposed Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.1 Interpolation of Missing G Pixels . . . . . . . . . . . . . . . . 27
3.2.2 Interpolation of Missing R/B Pixels . . . . . . . . . . . . . . . 29
3.3 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3.1 PSNR Comparison . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3.2 Subjective Comparison . . . . . . . . . . . . . . . . . . . . . . 30
3.3.3 Number of Required Memory Lines Comparison . . . . . . . . 36
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4 Architecture Design of CRISP-DS 39
4.1 Design Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2 Processor Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Dual-Stream Reconfigurable Interconnection Design . . . . . . . . . . 47
4.4 Reconfigurable Datapath design . . . . . . . . . . . . . . . . . . . . . 48
4.4.1 Local Memory RSPE . . . . . . . . . . . . . . . . . . . . . . . 48
4.4.2 Pixel-Based Operation RSPE . . . . . . . . . . . . . . . . . . . 53
4.4.3 Multiplication and Accumulation (MAC) RSPE . . . . . . . . . 56
4.4.4 Color Interpolation RSPE . . . . . . . . . . . . . . . . . . . . 56
4.4.5 ALU RSPE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.4.6 DownSampler RSPE . . . . . . . . . . . . . . . . . . . . . . . 63
4.4.7 ACC RSPE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.5 Summary of CRISP-DS . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5 Implementation 65
5.1 Design Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.2 Testing consideration . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.3 Implementation results . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.4 SOC system simulation . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.5 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.5.1 Hardware overhead for dual stream with context switch . . . . . 73
5.5.2 Comparison with Digital Signal Processor (DSP) . . . . . . . . 73
5.5.3 Comparison with CRISP . . . . . . . . . . . . . . . . . . . . . 75
5.6 Mapping Proposed Algorithm to CRISP-DS . . . . . . . . . . . . . . . 78
6 Conclusion 81

[1] W. Rabadi, R. Talluri, K. Illgner, J. Liang, and Y. Yoo, “Programmable DSP platform for digital still cameras,” Application Report, Apr. 2000.
[2] Jason C. Chen and Shao-Yi Chien, “Crisp: Coarse-grained reconfigurable image stream processor for ten mega pixels for digital still cameras,” Master Thesis, Department of Electronics Engineering, National Taiwan University, Taiwan, 2006.
[3] B. Bayer, “Color imaging array,” U.S Patent, vol. 3 971 065, 1976.
[4] J. E. Adams and J. F. Hamilton, “Adaptive color plane interpolation in single sensor color electronic camera,” U.S. Patent, vol. 5 629 734, May 1997.
[5] W. T. Freeman88, “Median filter for reconstructing missing color samples,” U.S Patent, vol. 4 724 395, 1988.
[6] D. R. Cok and Eastman Kodak Company, “Signal processing method and apparatus for producing interpolated chrominance valuse in a sampled color image signal,” U.S Patent, vol. 4 642 678, Feb 1987.
[7] R. Kimmel98, “Demosaicing: Image reconstruction from color ccd samples,”
IEEE Transactions on Image Processing, vol. 8, no. 9, pp. 1221–1228, Sep 1999.
[8] B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, “Color plane interpolation using alternating projections,” IEEE Transactions on Signal Processing, vol. 11, no. 9, pp. 997–1013, Sep 2002.
[9] S.-C. Pei and I.-K. Tam, “Effective color interpolation in CCD color filter arrays using signal correlation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 6, pp. 503–513, Jun 2003.
[10] W. Lu and Y. p. Tan, “Color filter array demosaicing: new method and performance measures,” IEEE Transactions on Signal Processing, vol. 12, no. 10, pp. 1194–1210, Oct 2003.
[11] R. Lukac and K.N. Plataniotis, “Normalized color ratio modelling for CFA interpolation,”IEEE Transactions on Consumer Electronics, vol. 50, no. 6, pp. 737–745, May 2004.
[12] X. Wu and N. Zhang, “Primary-consistent soft-decision color demosaicking for digital cameras,” IEEE Transactions on Signal Processing, vol. 13, no. 9, pp. 1263–1274, sep 2004.
[13] X. Li, “Demosaicing by successive approximation,” IEEE Transactions on Signal Processing, vol. 14, no. 3, pp. 370–379, mar 2005.
[14] K. Hirakawa and T.W. Parks, “Aaptive homogeneity-directed demosaicing algorithm,”IEEE Transactions on Signal Processing, vol. 14, no. 3, pp. 360–369, Mar 2005.
[15] W. Rabadi, R. Talluri, K. Illgne, J. Liang, and Y. Yoo, “Programmable DSP platform for digital still cameras,” in Proc. 1999 IEEE International Conference on Acoustics and Speech and Signal Processing (ICASSP99), Mar. 1999, pp. 2235 –2238.
[16] N. Nakano, R. Nishimura, H. Sai, A. Nishizawa, and H. Komatsu, “Digital still camera system for megapixel CCD,” IEEE Transactions on Consumer Electronics, vol. 44, no. 3, pp. 581–586, August 1998.
[17] H. Zen, T. Koizumi, H. Yamamoto, and I. Kimura, “A new digital signal processor for progressive scan CCD,” IEEE Transactions on Consumer Electronics, vol. 4, no. 2, pp. 289–296, May 1998.
[18] R. Zhou, X. Chen, F. Liu, J. He, T. Liao, Y. Su, J. Ye, Y. Qin, X. Yi, and Z. Hong, “System-on-chip for mega-pixel digital camera processor with auto control functions,”in Proceedings on 2003 International Conference on ASIC, Oct. 2003, pp. 894–897.
[19] D. Talla, C.-Y. Hung, R. Talluri, F. Brill, D. Smith, D. Brier, B. Xiong, and D. Huynh, “Anatomy of a portable digital mediaprocessor,” IEEE MICRO, vol. 24, no. 2, pp. 32–39, Mar.–Apr. 2004.
[20] Texas Instruments Inc., “TMS320DSC21—a high-performance, programmable, single chip digital signal processing solution to digital still cameras,”http://focus.ti.com/lit/ml/spry086/spry086.pdf, 2006.
[21] D. Talla, R. Austen, D. Brier, C.-Y. Hung, D. Huynh, D. Smith, B. Xiong, R. Talluri, and F. Brill, “TMS320DM310 — a portable digital media processor,” in Proceedings of IEEE HOT Chips, aug 2003, pp. 144–151.
[22] R. Hartenstein, “Reconfigurable computing: a new business model-and its impact on SoC design,” in Proceedings of Euromicro Symposium on Digital Systems, Design, Sept. 2001, pp. 103–110.
[23] B.K. Gunturk, J. Glotzbach, Y. Altunbasak, R.W. Schafer, and R.M. Mersereau, “Demosaicking: clolor filter array interpolation,” IEEE Signal Processing magazine, vol. 22, pp. 44–54, Jan 2005.
[24] Ting Chen, “A study of spatial color interpolation algorithms for single-detector digital cameras,” http://www-ise.stanford.edu/tingchen/main.htm, 1998.
[25] R.H. Hibbard, “Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients,” U.S Patent, vol. 5 382 976, 1995.
[26] http://r0k.us/graphics/kodak/.
[27] K.T. Gribbon and D.G. Bailey, “A novel approach to real-time bilinear interpolation,”in International Workshop on 2004 Second IEEE Electronic Design and Test and Applications, Janurary 2004, pp. 126–131.
[28] N. Nakano, R. Nishimura, H.Sai, A. Nishizawa, and H. Komatsu, “Digital still camera system for megapixel ccd,” IEEE Consumer Electronics, vol. 44, no. 3, pp. 581–586, Aug 1998.
[29] J.-S. Lee, Y.-Y. Jung, B.-S. Kim, and S.-J. Ko, “An advanced video camera system with robust af, ae, and awb control,” IEEE Consumer Electronics, vol. 47, no. 3, pp. 694–699, Aug 2001.
[30] http://www.cadence.com/.
[31] http://www.mentor.com/.
[32] http://www.synopsys.com/.
[33] http://www.tsmc.com/.
[34] http://http://www.cic.org.tw/.
[35] S. Agarwala, P. Koeppen, T. Anderson, A. Hill, M. Ales, R. Damodaran, L. Nardini, P. Wiley, S. Mullinnix, J. Leach, A. Lell, M. Gill, J. Golston, D. Hoyle, A. Rajagopal, A. Chachad, M. Agarwala, R. Castille, N. Common, J. Apostol, H. Mahmood, M. Krishnan, Duc Bui, Q.-D. An; P. Groves, N. Luong, N.S. Nagaraj, and R. Simar, “A 600MHz VLIW DSP,” Digest of Technical Papers of 2002 IEEE International Solid-State Circuits Conference(ISSCC 2002), vol. 2, pp. 38–390, Feburary 2002.