{'fraction_non_alphanumeric': 0.035513103110457994, 'fraction_numerical': 0.014776716466650339, 'mean_word_length': 4.006129955046997, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Low-Cost Imaging Method to Avoid Hand Shake Blur for Cell Phone Cameras

SUMMARY In this letter, a novel imaging method to reduce the hand shake blur of a cell phone camera without using frame memory is proposed. The method improves the captured image in real time through the use of two additional preview images whose parameters can be calculated in ad-vance and stored in a look-up table. The method does not require frame memory, and thus it can signiﬁcantly reduce the chip size. The scheme is suitable for integration into a low-cost image sensor of a cell phone camera.


Introduction
Hand-shaking usually introduces hand shake blur, which is detrimental to the performance of cell phone cameras. In the past, a deblurring algorithm that used a single blurred image was employed to obtain motion-blur-free images [1]. Recently, advances in image sensor have enabled fast image capture, and thus many algorithms now use multipleexposure images to generate higher-quality images. For example, Lee et al. [2] enhanced a low-exposed image via tonal correction using a blurred but bright image. A combined method has also been employed to post-process a pair of multiple-exposure images [3]. Choi et al. [4] and Tico et al. [5] obtained high-quality images by fusing a short-and long-exposed image, while Tsuda et al. [6] used multipleexposure images to restore a blurred image.
Compared to single image-based algorithms, multiple image-based algorithms can achieve better performance. However, algorithms that use multiple images must employ more than one frame memory when they are implemented by hardware. This will in turn lead to a significant increases in the chip area, especially for higher-resolution sensors. For example, in the TSMC 0.18 µm process, the frame memory for a full-sized image of a 3 M CMOS image sensor (CIS) (1/5 inch) will occupy an area of about 175 mm 2 , which is about ten times the size of the CIS. Thus, frame memory should not be used for decreasing the chip size and cost. In this letter, a low-cost imaging method to avoid hand shake blur that can be implemented by hardware without the need for frame memory is proposed. Two additional preview images are used to process a captured low-exposed image. Because the parameters of the two preview images can be calculated in advance, the processing of the captured image can be conducted in real time and frame memory can be removed.

Proposed Method
As shown in Fig. 1, the proposed imaging method uses three consecutive images: two preview images and a captured image. These images are exposed in long-short-long mode. The first preview image (P 1 ) was acquired under a long exposure time (T long ), while the second preview image (P 2 ) and first captured image (I 1 ) are acquired using a safe shutter speed (T short ). The brightness of P 1 is acceptable, although it is usually blurred because of the camera motion during the long exposure time. The motion blur of P 2 and I 1 is greatly reduced, but these images are low-exposed because of the short exposure time.
Considering that serious blur will occur with significant hand-shaking, we correct the low-exposed image I 1 , but do not restore blurred image P 1 , as shown in Fig. 1. First, we estimate the parameters of P 1 and P 2 , and then we obtain the color mapping relationship between the short-exposed and long-exposed images. The mapping is subsequently stored in a look-up table (LUT). The LUT-based color correction method is then used to correct the short-exposed I 1 . Because Copyright c 2012 The Institute of Electronics, Information and Communication Engineers the final output image has the brightness of P 1 and the crisp edge and texture of I 1 , it is a blur-free image of higher quality.
In contrast to the research conducted in the above-cited studies [2]- [6], we do not use P 1 to correct the color of I 1 directly. Such a step would require frame memory to buffer the data of I 1 when the color mapping function between P 1 and I 1 is estimated. Considering that P 2 and I 1 are temporally close, we use P 2 instead of I 1 to estimate the mapping function of the short-and long-exposed images. Because the parameters of P 1 and P 2 can be estimated in real time, the mapping can be obtained and stored it in a LUT before I 1 is processed. The color correction procedure for I 1 can be conducted in real time based on the pre-calculated LUT. Since both the mapping function estimation and color correction procedure are performed in real time, no frame memory is utilized, except for a small memory used to store the LUT.
2.1 Cumulative Histogram Matching P 1 and P 2 are first transferred from RGB to the YCbCr color space. The histogram and cumulative histogram of the images are then calculated in the YCbCr color space. For image P 1 , the histogram and cumulative histogram are calculated as follows: where h P 1 is the histogram of P 1 , L i is the amplitude of the luminance signal (Y-channel of YCbCr), W and H are the width and height of P 1 , respectively, n i is the number of the pixels (i is the gray level), and C P 1 is the cumulative histogram of P 1 . The cumulative histogram C P 2 [L j ] of P 2 can be calculated in the same manner. The image with the cumulative histogram of C P 2 can be mapped to an approximation of the desired levels of the image with the cumulative histogram of C P 1 . Through this approximation, we can determine the mapping function F between two images as follows [7]: As shown in Fig. 2, the mapping process consists of three steps: S 1 , S 2 , and S 3 . First, through the L j of P 2 , we can find the corresponding C P 2 [L j ]. The approximation of C P 2 [L j ] in C P 1 [L i ] can then be determined. Finally, through the C P 1 [L i ] of P 1 , the mapped luminance L i is obtained.
Because the captured image I 1 is temporally close to P 2 , its histogram is very similar to that of P 2 . Once the mapping F between the images P 2 and P 1 is determined, F can be applied to modify the histograms of the captured image I 1 . The color correction is as follows, where L k is the original luminance level of I 1 , and L k is the corrected luminance of the output image.

LUT-Based Histogram Matching
To implement the mapping of Eq. (4) in a straightforward manner, a 2-D LUT is constructed. One dimension of the 2-D LUT is L k , while the other dimension is L k , as shown in the upper portion of Fig. 3. However, the straightforward LUT (SLUT) needs to store both the L k and L k , and must also remember the mapping function F between L k and L k . Thus, the SLUT is difficult to implement and uses more memory.
Here, we propose an improved LUT (ILUT). Considering that the maximum possible number of the luminance levels of both I 1 and the corrected I 1 , i.e., L k and L k , is 255, we used L k as the address of L k directly, as shown in the bottom portion of Fig. 3. Thus, memory is conserved and methods to store the mapping function do not need to be considered. Furthermore, several optimization algorithms are proposed. First, the cumulative histograms are scaled by a scaling factor of 255. After the scaling, the ranges of C P 1 and C P 2 are changed from [0, 1] to [0, 255], and are denoted as C P 1 and C P 2 . When C P 2 is mapped into C P 1 and no mapping value exists for cumulative histogram C P 1 at the integer location in [0, 255], the nearest rule is used to generate the output. Second, a strategy to address the problem of having n values of L P 1 mapping to the same C P 1 was employed. For more than five L P 1 with the same C P 1 , the middle value of the values is used as the final mapping; otherwise, if there are less than five L P 1 with the same C P 1 , we use the maximum L P 1 as the final mapping.
After the optimizations, the SLUT is transformed into the ILUT, which is a 1-D table with the address of L I 1 and an output of L I 1 (L P 1 ). The histogram matching is simplified to retrieve a LUT in an on-chip memory.

Timing Design
As shown in Fig. 4, the pipelining framework mainly consists of three stages: the calculation of cumulative histogram of P 1 , the calculation of cumulative histogram of P 2 , and the LUT-based color correction of I 1 .
In Fig. 4, we performed RGB2YCbCr modules via fixed-point calculation. The cumulative histograms of P 1 and P 2 are also calculated through fixed-point operation. Stages I and II are completed with the latencies of L 1 and L 2 , respectively. Both of these latencies are smaller than the interval between the two images. In the third stage, the cumulative histogram calculation, RGB2YCbCr and YCbCr2RGB modules are used in the same manner with as in stage I. The LUT-based color correction is simplified to on-chip memory access since the ILUT is used. The latency of this stage, L 3 , was also smaller than the interval between the two images and did not affect real time implementation.
Because we use the parameters of P 1 and P 2 to process I 1 , the proposed method does not require a frame memory to buffer I 1 . Furthermore, because both stages I and II can be conducted in real time, no frame memory is needed for this system.

Simulation and Results
To verify the similarity between the histogram of P 2 and I 1 , we took 10 consecutive frames by a hand-held camera (shut- ter speed of 1/64 s, in a dimly lit office) and calculated their histograms. As shown in Fig. 5, almost all of the histograms of the 10 frames were very similar.
One test chart and two different scenes (a folk museum and a vending machine) were used to validate the color correction process; the results are illustrated in Fig. 6. For subjective evaluation, Figs. 6 (m)-(o) shows that the resulting images of our method are as bright as Fig. 6 (a)-(c) and are as crisp as Fig. 6 (j)-(l).
For an objective evaluation, the CIELAB color distortion metric was used to determine the the efficiency of the LUT-based color correction algorithm. This metric may be expressed as follows: where ΔL * , Δa * , and Δb * are the differences in L*a*b* color space, and W and H are the width and height of the test image, respectively. According to the standard, the distortion is barely perceptible when ΔE is smaller than 3, while it is perceptible but acceptable when ΔE ranges from 3-6. In our simulation, the ΔE between the output images ( Fig. 6 (m)-(o)) and the reference images ( Fig. 6 (a)-(c)) are 0.23, 0.89, and 0.35, respectively. To evaluate the degree of blurring, a modulation transfer function (MTF) [8] was employed as a metric. Both horizontal and vertical edges were used to estimate the MTF; the results are shown in Table 1. A comparison to the values of MTF50 revealed that the difference between the corrected images and reference images is very small, while the difference between the blurred images is great. These findings show that the proposed algorithm greatly decreases the blur level of a blurred image much and yields results that are similar to the reference images.
Different blur reduction schemes and their manner of implementation, the numbers of frames, and the final memory usage are listed in Table 2. For conventional multipleexposure-based algorithms, one or more frame memories are needed if the algorithms are implemented by hardware in real time. Because the latencies of L 1 , L 2 , and L 3 are only a couple of clocks and the interval between the two images is longer than 1 ms, the proposed method can be im- Fig. 6 Test images and the results of LUT-based color correction: (a)-(c) are reference images; (d)-(f) are blurred images previewed with long exposure time; (g)-(i) are low-exposed images previewed with short exposure time; (j)-(l) are lowexposed images captured with short exposure time; and (m)-(o) are the output images obtained by our proposed method.  plemented by hardware in real time and requires no frame memory. The only memory usage is for the ILUT, which occupied 256 bytes. However, this size is negligible and does not significantly increase the chip area.

Conclusions
A novel and efficient imaging method that can be implemented by hardware without the use of frame memory is proposed. The proposed scheme is suitable for low-budget cameras that have constraints with respect to chip area and cost.