SINGLE IMAGE SUPER-RESOLUTION AND COMPLEXITY-QUALITY TRADE OFF

Single Image Super-Resolution is a challenging task that aims to enhance the quality of the image from low resolution to high resolution. Superresolution techniques can also be used as image lossy compression-decompression technique where low-res image is transmitted and decoded into high-res version at the received end. Various methods have been introduced till date, including algorithmic interpolation methods and deep learning methods. The algorithmic methods such as bilinear interpolation, bicubic interpolation, nearest neighbor and others provide fast processing while deep learning methods provide better quality of superresolution image. We analyzed the time required to process images using various methods and compared them with the perceptual quality along with the variation in perceptual quality and PSNR metrics. Keywords— Super-Resolution, Image Processing, Deep Learning, Image Super-Resolution, Machine Learning

INTRODUCTION On the basis of number of pixels in a digital image, a digital image can be a low-resolution (LR) or a high-resolution (HR). This categorization is subjective and there is no specific boundary defining the separation of low-resolution images from high-resolution images, however we will be using this idea as a means to compare two or more versions of same image. Higher-resolution images have more pixels than the low-resolution image, thus they contain more information and finer details about their subject [1]. Capturing low-resolution images is easier and cost effective, but trades off with quality. High-resolution images have better quality but need more resources in terms of camera quality, storage (memory), network bandwidth, etc. If we can reduce the number of pixels required to store a digital image, without losing on quality, it would reduce the overall cost of capturing, storing and transferring the images.
Super-resolution (SR) is the method of upscaling and improving the textures, details within an image. A lowresolution image is taken as an input and then it is upscaled to a higher resolution as the output. The details in the highresolution output are filled in where the details are essentially unknown. It is the highly challenging task of estimating a high resolution (HR) image from its low-resolution (LR) counterpart. Single Image Super-Resolution (SISR) received substantial attention from within the computer vision research community and has a wide range of applications. Various interpolation algorithms have been devised to upscale lower resolution images into HR images. These algorithms are widely popular and have found their way as necessary tools included in most of the image processing libraries. Some of the popular algorithms are bicubic interpolation [3], bilinear interpolation [3], Lanczos [3,5], nearest neighbor interpolation [3], etc. We will briefly study these methods in Section 3.
Since introduction of SRCNN [7], deep convolution neural network (CNN) approaches have brought prosperous development. Various network architecture designs and training strategies have continuously improved the SR performance, especially the Peak Signal-to Noise Ratio (PSNR) value. However, these PSNR-oriented approaches tend to result in over-smoothed results without sufficient high-frequency details, since the PSNR metric fundamentally disagrees with the subjective evaluation of human observers [5]. One of the turning points in the way of pursuing visually more accurate results is SRGAN [9] model. The basic model is built with residual blocks and optimized using perceptual loss in a GAN [12] framework. With all these techniques, SRGAN significantly improves the overall visual quality of reconstruction over PSNR-oriented methods. It also uses deep residual blocks with skip-connection and diverges from MSE as the sole optimization target. It defines a novel perceptual loss using high-level feature maps of the VGG [11] network combined with a discriminator that encourages solutions perceptually hard to distinguish from the HR reference images, which is unique from previous works.
The objective is to improve the low-resolution image to be as good (or better) than the target, known as the ground truth, which in this situation is the original image we downscaled into the low-resolution image. However, there remains a considerable difference between SRGAN generated results and the ground-truth (GT) images. However, one limitation that super resolution models suffer with i.e. introduction of distortions called artifacts is still present in SRGAN.
In the ESRGAN [12] model, the key components of SRGAN have been revisited and the model was improved in three aspects. First, the network structure by introducing the Residual-in-Residual Dense Block (RDDB) [11], which is of higher capacity and easier to train, has been improved. Also, Batch Normalization layers have been removed and residual scaling has been used instead along with smaller initialization to facilitate training a very deep network. Second, the discriminator using Relativistic Average GAN [10], which learns to judge the image's reality relative to other image rather than the realness of the image, has been improved. Third, an improved perceptual loss by using the VGG features before activation instead of after activation as in SRGAN, is used.

II.
BRIEF OVERVIEW OF INTERPOLATION METHODS Interpolation is a method of constructing new data points a) Nearest neighbor [3] In this method, we expand the image to required size and fill in the gaps by duplicating the value of known pixel which is nearest to the required pixel.
I(x,y) = nearestNeighbor(x, y) This method is very simple and very fast. However, it introduces jagged edges and pixelated images. It is only good for simple images containing simple object shapes. Detailed images with curves and shades of intensity suffer from loss of quality when processed with nearest neighbor technique.

b) Bilinear Interpolation [3]
In the field of numerical analysis, bilinear interpolation is the extension of linear interpolation in which we interpolate a bivariate function f(x,y) over given set of data points first along one axis and then the other axis. This method is also called bilinear filtering or bilinear texture mapping.
It is assumed that we know the value of f at the four points 11 = ( 1 , 1 ), 11 = ( 1 , 2 ), 21 = ( 2 , 1 ), 22 = ( 2 , 2 ). We can write the solution to the interpolation problem as f( , ) = 0 + 1 x + 2 y + 3 xy Where the solution of the following linear system gives the value of the coefficients

c) Bicubic Interpolation [3]
Bicubic interpolation is an extension of cubic interpolation. It is slower than bilinear interpolation and nearest neighbor interpolation and is used when speed is not a great issue. However, it is still faster than deep learning methods. Bicubic interpolation considers 16 pixels (4x4 grid) and results in lesser artifacts and smoother image.
Suppose the function values and the derivatives , and are known at the four corners (0,0), (0,1), (1,0) and (1,1) of the unit square, the interpolated surface can be written as The interpolation problem consists of determining the 16 coefficients (aij). The equations for those can be found in detail at ref. 3.

III.
BRIEF OVERVIEW OF DEEP LEARNING METHODS With the introduction of SRCNN [4] (deep convolutional neural network for super-resolution) in 2014, the field of super-resolution achieved a breakthrough. Several deep learning methods including residual networks and GANs have been explored since then as the solution to the ill-posed problem of super-resolution. The power of neural networks to automatically figure out the transformation, given the dataset has proved to be incredibly useful to achieve higher quality SR images having greater amount of detail, sharpness, and lesser number of artifacts. Let's go through some of the remarkable approaches below-

a) SRCNN [4]
SRCNN was a breakthrough in the field of super-resolution, unlocking the potential of deep convolutional neural networks for super-resolution applications and other abstract computer vision problems. SRCNN architecture was simple in terms of computational power and resources required, yet effective in quality. It had two convolution layers, the first one extracted the features from the low-resolution image, while the second layer maps these feature maps to high resolution. The architecture of SRCNN is shown in Fig. 1.
It used the PSNR as loss function. PSNR (Peak Signal to Noise Ratio) is based on mean square error (MSE) between the ground truth (HR image) and image obtained through super-resolution (SR image). Reference 5 showed that distortion measures such as PSNR and SSIM vary greatly from the perceptual quality (human perception of image quality) of images.

b) Residual Dense Network (RDN) for Super-Resolution
Yulun et. al. proposed RDN for SISR [14] which used residual blocks to allow more layers in the network. In residual blocks, residuals from the previous layers are added to output of later layers so as to preserve the outputs from the previous layers. It was also based on PSNR maximization by minimizing MSE between HR (ground truth) and SR image (RDN output).   [14] c) SRGAN [6] SRGAN was the first architecture to explore the use of generative adversarial networks for SISR applications. It also used perceptual index approach instead of distortion (PSNR) based approach. The improvement in quality of SR images achieved using this architecture created a new benchmark for all future works. It used 5 residual blocks and a skip connection from first activation layer in the generator to the first batch normalization (BN) [5] layer after the residual blocks ( fig. 3). This BN layer is followed by two up-sampling blocks to achieve 4x up-sampling. The output of the generator is then fed to a discriminator which has a VGGstyle architecture containing stack of convolution activation and pooling layers. The discriminator uses VGG feature loss as proposed in refs. 10 as perceptual loss and predicts if the output of generator appears to be a real image or a fake image.

d) ESRGAN [12]
Enhanced Super-Resolution Generative Adversarial Network is an improvement over SRGAN. It uses Residual-in-Residual Dense Blocks (RRDB) [12] and has a greater number of layers that SRGAN. It removes the batch normalization layer [5] and replaces discriminator with relativistic discriminator [13]. Hence, it is based on Relativistic Average GANs (RaGAN) [6]. It focuses on a balance between perceptual index and distortion metrics such as PSNR, SSIM, etc. through network interpolation [12]. The authors claim that ESRGAN produces visually more perceptive super resolution image than its predecessor SRGAN. The RaGAN is also harder to train than simple discriminator used by SRGAN.

IV. METHODOLOGY AND RESULT
We used pre-trained Residual Dense Network (RDN) and ESRGAN trained on Div2K dataset. The RDN network we used, is designed for 2x up-sampling, so we used two RDN networks back-to-back to get 4x upscaled images. The interpolation methods and ESRGAN are tuned to produce 4x up-sampled images. We fed random 50 images from Set5, Set14, Urban 100, BSD 100, Sun-Hays 80 datasets.
We recorded the time taken to generate up-sampled images by each method for each image and then performed box plot analysis to compare how much the time taken, perceptual quality and PSNR obtained vary from model to model. We also verified the box plot analysis by visually verifying the generated up-sampled images.
The hardware we used had Intel Xeon 2.30 GHz Dual Core with 11 Giga bytes of RAM. The time taken to compute the results will vary greatly from processor to processor as it is a function of processor throughput, but the gain in perceptual quality will not be much affected.
We recorded the time in seconds as a measure of timecomplexity of super-resolution methods, and perceptual loss from VGG-19 (trained on ImageNet) as inverse of perceptual quality (lower loss means higher perceptual quality). by different methods to process 50 images, we can see that the time taken by interpolation methods is negligible in comparison to deep learning methods. We also observe that 4x up-sampling using dual RDN model back-to-back is a lot slower and is twice as large than 4x up-sampling ESRGAN model.
From Fig 5(b) and 5(c), we can see that nearest neighbor method performs bad in terms of both perceptual quality and PSNR. ESRGAN achieves highest perceptual quality (lowest perceptual loss) and lower PSNR than RDN network. Hence, we can deduce that ESRGAN provides the best in terms of perceptual quality and is better than RDN in terms of time.
From Fig. 5(a), (b) and (c), we can infer that interpolation methods are better in terms of time complexity but the deep learning methods can achieve better results in terms of quality.
In Fig. 6, we provide some of the up-sampled images along with low resolution (LR) and ground truth (HR) images for visual comparison of perceptual quality. The findings from box plot analysis of perceptual quality can be verified from visual comparison of images. In these images, we can see that deep learning methods achieve more detailed up-sampled images than interpolation techniques, and produce smoother results with lesser artifacts.
The original data points used to create box-plots is provided in appendices A, B and C. The values can deviate slightly with each execution.

V. CONCLUSION
Here, we reviewed existing super-resolution techniques and showed that deep learning methods are quite costly in terms of time and hardware resources required, but also give better results. We can use the traditional interpolation methods when time-complexity is of utmost importance and very high quality of up-sampled image is not required. When more emphasis is on the quality of resultant images, then we can use state-of-the-art deep learning methods but suffer in terms of time and resources required.
VI. FUTURE WORK We performed our analysis on limited dataset and limited number of super-resolution techniques. In future, we hope that the analysis will be expanded to other techniques as well and the performance of these techniques be evaluated on different hardware and operating systems to find a more substantial evidence of the complexity-quality tradeoff. VIII.