Solving Inverse Computational Imaging Problems using Deep Pixel-level Prior
Akshat Dave Anil Kumar Vadathya Ramana Subramanyam Rahul Baburajan Kaushik Mitra




































Journal of  Class Files, Vol. XX, No. XX, April 20XXDave et al.: Solving Inverse Computational Imaging Problems using Deep Pixel-level Prior




























Solving Inverse Computational Imaging Problems 
 using Deep Pixel-level Prior
    Akshat Dave, Anil Kumar Vadathya, Ramana Subramanyam, Rahul Baburajan, Kaushik Mitra, Member,  IEEE 
Akshat Dave is with the Department of Electrical and Computer Engineering, Rice University, Houston, TX, USA (e-mail: akshat.dave@rice.edu).
Anil Kumar Vadathya, Rahul Baburajan and Kaushik Mitra are with the Computational Imaging Lab, Indian Institute of Technology (IIT) Madras, Chennai, India.
Ramana Subramanyam is with Insight Centre for Data Analytics, Dublin, Ireland.
    December 30, 2023
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================







Signal reconstruction is a challenging aspect of computational imaging as it often involves solving ill-posed inverse problems. Recently, deep feed-forward neural networks have led to state-of-the-art results in solving various inverse imaging problems. However, being task specific, these networks have to be learned for each inverse problem. 
On the other hand, a more flexible approach would be to learn a deep generative model once and then use it as a signal prior for solving various inverse problems. 



We show that among the various state of the art deep generative models, autoregressive models are especially suitable for  our purpose for the following reasons. First, they explicitly model the pixel level dependencies and hence are capable of reconstructing low-level details such as texture patterns and edges better. Second, they provide an explicit expression for the image prior which can then be used for MAP based inference along with the forward model. Third, they can model long range dependencies in images which make them ideal for handling global multiplexing as encountered in various compressive imaging systems. 

We demonstrate the efficacy of our proposed approach in solving three computational imaging problems: Single Pixel Camera (SPC), LiSens and FlatCam. For both real and simulated cases, we obtain better reconstructions than the state-of-the-art methods in terms of perceptual and quantitative metrics.



Image reconstruction is a challenging aspect of computational imaging cameras as it often corresponds to solving ill-posed inverse problems. Using deep neural networks to learn the inverse mapping from sensor measurements to the image offers state-of-the-art reconstructions. But this approach makes the network tailored to a particular forward model. On dealing with multiple configurations of the forward model, it is expensive to have a separate network for each configuration. Hence, we use deep learning to learn a single image prior which can be utilized for inverting any arbitrary forward model. Deep generative models are quite powerful in modelling natural image statistics. In particular, deep autoregressive models provide state-of-the-art performance, in terms of log likelihood scores, by modelling tractable densities over the image manifold. In this work, we employ one such learned deep autoregressive network (PixelCNN++) as the image prior for solving multiple inverse problems using Maximum A Posteriori inference. We demonstrate how the images reconstructed from our approach have better pixel-level consistencies, as compared to the existing deep generative model approaches. We test our approach on three computational imaging setups: Single Pixel Camera, LiSens and FlatCam with real and simulated measurements. We obtain better reconstructions than the state-of-the-art methods in terms of both perceptual quality and quantitative metrics.



Most of the computational imaging regimes capture the visual world in terms of coded or compressive measurements. Reconstructing high quality images from these measurements is often ill-posed and requires good data priors. In this work, we propose a simple yet versatile framework for solving any such inverse problem using a prior learned on natural image statistics. We show that with a deep autoregressive model as the prior, one can reconstruct images which have better pixel-level consistencies, as compared to the existing deep auto-encoder based approaches. We test our approach on three computational imaging setups: Single Pixel Camera, LiSens and FlatCam with real and simulated measurements. We obtain better reconstructions than the state-of-the-art methods for these setups, in terms of both perceptual quality and quantitative metrics such as PSNR and SSIM. We also show how randomly dropping the update of some pixels in every iteration helps in a better image reconstruction.    





Inverse problems, compressive image recovery, deep generative models, lensless image reconstruction,  autoregressive models, MAP inference.





















§ INTRODUCTION


Computational imaging systems enable us to extract much more information out of the visual world as compared to the traditional imaging systems. This is achieved by jointly designing optics, to encode the desired signal information, and algorithms to reconstruct the signal back from those measurements. Signal reconstruction corresponds to inverting the forward model used in acquiring the measurements. Hence, reconstruction algorithms for different computational imaging devices amount to solving different inverse problems. Solving these inverse problems becomes challenging as they are often ill-posed. For  compressive imaging setups such as Single Pixel Camera (SPC) <cit.>, <cit.>, high speed imaging <cit.>,  <cit.> and compressive hyper-spectral imaging <cit.>, the reconstruction becomes ill-posed as the number of measurements is quite less than the signal dimension. 

Generally, for solving an ill-posed problems, we need to incorporate the prior information about the signal to be reconstructed. Traditionally these priors are either analytically derived or hand-crafted based on the observations. For example, sparsity of image gradients <cit.>, sparsity of coefficients in wavelet and DCT domain <cit.> etc. have been used for solving inverse imaging problems. However, the underlying data distribution may not precisely follow these analytic priors leading to poor solutions in challenging scenarios. Dictionary learning <cit.> methods being data driven are an improvement over these analytic priors. However, being limited by patch size they cannot account for long range dependencies which are necessary for handling global multiplexing in case of compressive image reconstruction.





On the other hand, deep learning based reconstruction algorithms recently have led to state-of-the-art results in solving such ill-posed problems in computational imaging <cit.>, <cit.> <cit.> <cit.>. These approaches typically learn an inverse mapping from measurements to the signal by minimizing reconstruction loss on a set of training examples. However, this kind of training, popularly known as discriminative learning, makes the network task specific. 

Furthermore, we need to retrain the network for various parameter settings of the forward model. For example, for every new setting of measurement rate and sensing matrix in SPC, we need to relearn the network parameters.  
Instead of having to design/retrain a different network for each task and parameter setting, it would be more efficient to have a generalized framework which can be used for solving various inverse problems.










A more flexible approach would be to learn the natural image statistics using a generative model and use it for solving various inverse problems. 

Recently, deep generative models especially using autoregressive framework <cit.> have led to state-of-the-art performance in modeling natural image manifold. Autoregressive models factorize the image distribution as a 2D directed causal graph and hence model it as a 2-D sequence where current pixel's distribution is conditioned on the causal context. 

By employing deep neural networks for summarizing the causal context, autoregressive models excel at capturing long range dependencies in images. Also, being a pixel level model it explicitly accounts for higher order correlations like texture patterns, sharp edges, etc. within a neighbourhood. Thus, these models are capable of generating visually convincing and crisp images <cit.>. Examples of deep autoregressive image models are recurrent image density estimator (RIDE) <cit.>, pixel recurrent neural networks (PixelRNN) and its CNN equivalent (PixelCNN) <cit.> and PixelCNN++ <cit.>. 

We show that deep autoregressive generative models are ideally suitable for solving various computational imaging problems for the following reasons. First, it explicitly models the distribution of each pixel in relation to its causal neighbor. Thus, when used as an image prior, this explicit pixel dependency modeling helps it to better reconstruct low level details without artifacts (see Figure <ref>). Second, this framework gives us an explicit expression for the image prior, which can be used for doing MAP inference. Moreover, the entire framework is differentiable, which is amenable for gradient based inference. Third, its ability to capture long range dependencies in images makes them ideal for handling global multiplexing in compressive imaging setups. 
Given these advantages with deep autoregressive models, we use it for solving various computational imaging problems such as - Single Pixel Camera (SPC) <cit.>, Line Sensor (LiSens) <cit.> and lensless imaging - FlatCam <cit.>. Our results demonstrate that we perform better than the current state-of-the-art methods in both traditional and learning based approaches. 

In summary we make the following contributions: 

    
  * We propose a versatile approach which employs the same learned prior model for solving various computational imaging problems. 
    
    
  * We propose to use a deep autoregressive model, PixelCNN++, as an image prior. The autoregressive nature of this prior ensures pixel-level consistencies in the reconstruction and hence provides better quality than using latent representation based models such as OneNet <cit.> as shown in Figure <ref>.
    
    
  * We utilize back-propagation to the inputs for obtaining tractable estimates of the prior gradients and employ them for solving inverse problems using MAP inference.
    
    
  * We observe that randomly dropping the gradient updates for a certain percentage of pixels at every iteration helps in reconstructing the texture better. We analyze the effect of this pixel dropout ratio on the quality of reconstructions.
    
    
  * We demonstrate better reconstructions than the existing state-of-the-art methods for three computational imaging problems: Single Pixel Camera, LiSens, and FlatCam. 







































§ RELATED WORK














Compressive imaging Single Pixel Camera (SPC) <cit.> is a classic example of compressive imaging. It uses a programmable digital micro-mirror device (DMD) array to multiplex the scene on to a single photodetector. Using different settings on the DMD, we can sequentially acquire a set of measurements. Thus, scene at full resolution is reconstructed from much less than 100% measurements. Compressive imaging systems pose a viable solution for high resolution imaging in non-visible parts of the spectrum where full frame sensors are very expensive. 


The measurement bandwidth of the SPC is limited by the operating speed of the DMDs (Tens of kHz for commercially-available units). With this speed, SPC cannot be extended for high resolution video sensing. On one end, we have exorbitant full frame sensors (Nyquist sampling) for high resolution imaging in non visible bands, and on the other, we have SPC, an inexpensive compressive sensing setup but with low measurement rates. Wang et al. <cit.> propose LiSens - Line Sensor based compressive camera which lies midway between these two imaging extremes. Each pixel in the line sensor is mapped to a row in DMD array. Thus, unlike SPC, where the whole scene is multiplexed, here only rows of the scene are multiplexed. 

Lensless imaging FlatCam <cit.> and DiffuserCam <cit.> are novel imaging systems which get rid of the conventional lens optics. Instead, they use amplitude and diffuser mask respectively to encode light coming from different parts of the scene onto the sensor. As a result, information localized at a point in the scene gets spread throughout the sensor, making priors essential for accurate recovery of the image. These works use traditional reconstruction algorithms such as Total Variation norm and Tikhonov regularization which are quick but do not provide natural looking reconstructions. 

Reconstruction with analytical priors Many algorithms have been proposed for compressive image reconstruction. Typically, reconstruction algorithms use l_1 regularization, exploiting the sparsity of spatial gradients in natural images. Total Variation (TV) minimization prior <cit.> is the most commonly used reconstruction algorithm based on this sparsity. Chengbo et al. <cit.> propose an efficient augmented Lagrangian based TV minimization for CS reconstruction. Recent approaches involving compressive architectures such as fpa-cs <cit.>, LiSens <cit.>, and video CS <cit.>, demonstrated successful results with TV minimization prior. However, at lower measurement rates, reconstructions suffer from the piece-wise smooth modeling of TV prior and results tend to be blocky, as is noted by recent works <cit.>. Metzler et al. <cit.> propose a denoiser based CS reconstruction algorithm. Specifically, use a Gaussian denoiser with approximate message passing algorithm (D-AMP). At very low measurement rates, the denoiser tends to result in overly smooth images as is recently shown by Dave et al. <cit.>, Kulkarni et al. <cit.>.

Data driven CS reconstruction Duarte et al. <cit.> propose an approach for simultaneous learning of the sensing matrix and dictionary atoms. Due to the small patch size of the atoms, their usage for compressive image reconstruction is limited to local multiplexing, unlike the actual SPC involving global multiplexing of the scene. Reconstruction algorithms using convolutional neural networks (CNNs) typical take input as measurements from an image patch and try to output the image back by minimizing the reconstruction loss. Kulkarni et al. <cit.> proposed ReconNet, Yao et al. <cit.> proposed DR^2-Net having residual connections for reconstruction. Although these approaches lead to a non-iterative and hence faster inference, being task specific, they only work for the fixed settings of the sensing matrix and measurement rates used for training. Changing the settings requires retraining the architecture which is not very appealing. Also, being patch-wise, they also fail to account for global multiplexing in SPC. 

Deep generative models
With the success of deep neural networks, there have been multiple works proposing deep generative models, which explicitly or implicitly try to model the distribution of natural images. For example, latent representation models like adversarial networks, GAN by Goodfellow et al. <cit.>, variational auto-encoders by Kingma et al. <cit.> and autoregressive models like RIDE by Theis et al. <cit.>, PixelRNN/CNN by Oord et al. <cit.>, PixelCNN++ by Salimans et al. <cit.>. GANs learn to transform samples from a Gaussian distribution to a sample in the natural image manifold via a generator network, which is trained with an adversarial learning framework involving a discriminator network. VAEs are a probabilistic framework of autoencoders that learn to encode and decode the images from a distribution. 

Autoregressive models factorize an image as a 2D directed graph by conditioning the current pixel x_i's distribution on the pixels before it as in a raster scan x_<i. Modeling this conditional density is analogous to sequence modeling and initial methods proposed to use spatial 2D recurrent neural networks, given their efficacy in modeling sequences. RIDE by Theis et al. <cit.> uses 2D Long Short Term Memory (LSTM) units called Spatial-LSTMs for modeling the causal context x_<i, and GSMs for parametrizing the distribution. PixelRNN by Oord et al. <cit.> uses a much complex architecture using LSTMs and residual connections to better handle the causal context. Importantly, it models the conditional density as a discrete distribution with x_i ∈{0, 1, … 255}. PixelRNN has resulted in state-of-the-art negative loglikelihood (NLL) scores. However, due to the sequential nature of distribution modeling, both training and sampling are computationally demanding with the runtime as O(N), where N is the total number of pixels. Oord et al. proposed PixelCNN which is a convolutional version of PixelRNN. This led to an improvement in
the training time by a large factor at the cost of slight loss in the accuracy as with convolutions we can now only capture bounded context. Salimans et al. <cit.> proposed PixelCNN++, which builds on PixelCNN by employing a discretized mixture of logistics for modeling the distribution, and using drop-out regularization, and additional skip connections. It improves on the NLL score over PixelRNN on the CIFAR dataset leading to state-of-the-art results. 



Deep image priors When solving linear inverse problems using the alternating direction method of multipliers (ADMM) algorithm, Venkatakrishnan et al. <cit.> observed that it results in two decoupled optimizations. The first one enforces the data prior while the second enforces data fidelity to the observation. The first step can be thought of as a denoising problem, thus, a denoiser can be employed to solve this step thereby avoiding the need for an explicit image prior. Venkatakrishnan et al. <cit.> use denoisers like BM3D <cit.> in ADMM setting for image restoration. Inspired by this, recent methods propose learning-based proximal operators for the denoising step of ADMM. OneNet by Chang et al. <cit.>, CNN denoiser by Zhang et al.<cit.>, Meinhardt et al.  <cit.> . In this work, we compare our explicit natural image prior based MAP inference with the learned proximal operator of OneNet. Our evaluations show that our results are superior to OneNet. It is important to note that OneNet's proximal operator uses adversarial loss <cit.> which is known to result in sharper recovery of details.

In this paper, we extend upon our previous work, Dave et al. <cit.> (RIDE-CS), where we used recurrent image density estimator (RIDE) for CS reconstruction. We observed that the sequential nature of recurrent networks in RIDE makes it too slow for inference and training (computational cost is proportional to the image size). Also, in our experiments, the two layer RIDE fails to yield results comparable to recent approaches like OneNet <cit.>. Here, we explore sophisticated deep autoregressive models which are order faster than RIDE-CS for both training and inference. We apply the deep autoregressive model based inference to recent frameworks in computational imaging like LiSens <cit.> and FlatCam <cit.>. We enhance the inference algorithm by incorporating the augmented Lagrangian method when necessary. 
In addition, we improve texture recovery using pixel-wise stochastic gradient updates.  




§ INFERENCE WITH DEEP AUTOREGRESSIVE MODELS





 §.§ Problem Formulation

Consider  to be a n× n matrix corresponding to a natural image and f to be a linear transformation corresponding to the forward model of a computational camera. The measurements obtained  can be written as = f(). Our goal is to reconstruct back the image  from the measurements . 

Discriminative networks learn the inverse mapping = g() by modelling g as a deep neural network and minimizing the reconstruction error on a set of training examples {_i,_i}. Hence, the inverse mapping is implicitly dependant on the forward model f. Dealing with reconstructions for multiple forward models would require learning separate networks for each model which can be expensive. 

For our generative approach, we model the distribution of natural images p() using a deep autoregressive model.  
We formulate the inverse problem as MAP inference. Hence, the estimated image 𝐗̂ can be written as 

    𝐗̂   = _ log(p(|))
       = _ ( log(p(|)) + log(p()))

The likelihood term p(|) varies for different imaging systems based on the forward model but the image prior p() remains the same. Thus, we need to learn the prior only once for all the problems.



 §.§ Forward Models

Let the n^2 × 1 column vector 𝐱 represent the rasterized version of the n × n image matrix  i.e. 𝐱≜ vec() by taking pixels row by row. The forward models that we consider in this work are as follows:


  §.§.§ Randomly Missing Pixels

Here, we randomly set certain number of pixels in an image to by missing, by setting their values to zero. Hence,  i.e. the vectorized version of the resultant image can be written as 

    = 𝐦∘

where ∘ denotes the Hadamard product and 𝐦 is a Bernoulli random vector. The above equation can also be expressed in a matrix-vector multiplication form as :  

    = 𝐌

where 𝐌 is a sub-sampling matrix. 





  §.§.§ Single Pixel Camera

In SPC <cit.>, the DMD array optically multiplexes the scene onto a single pixel sensor. By changing the orientation of the array, we will get different multiplexing patterns, which results in different measurements. If  is the vector of m single pixel measurements from SPC and Φ is the m × n^2 compressive sensing matrix, then we have the forward model as:

    = Φ.




  §.§.§ LiSens

In Lisens <cit.>, the 2D image of the scene formed on the DMD plane is mapped onto a 1D line-sensor which essentially captures the 1D integral of the 2D image (along rows or columns). If Y is the m × n matrix formed by stacking m line sensor measurements from Lisens and Φ is the m × n sensing matrix, then we have

    = Φ




  §.§.§ FlatCam

FlatCam <cit.> replaces the lens system by a coded amplitude mask close to the sensor. For ease of calibration, this mask is designed to be separable, i.e., it can be written as an outer product of 2 one dimensional patterns. Neglecting the diffraction effects, it was shown in <cit.> that using such a mask, the m × m measurements  obtained on the FlatCam sensor can be written as 

    = Φ_LΦ_R^T

where Φ_L and Φ_R are m × n matrices corresponding to 1-D convolution of the scene  along the rows and columns respectively. 




 §.§ Deep autoregressive model

Here we model the dependencies between pixels using a directed probabilistic chain. The pixel x_i depends on all the pixels before the index i in , which we denote as 𝐱_<i. Hence the joint distribution over the pixels in the image can be factorized as 


    p() = p(x_1,x_2,…,x_n^2) = ∏_i = 1^n^2 p(x_i|_<i)


In this work, we use state-of-the-art autoregressive generative model, PixelCNN++ <cit.>. Here, the context 𝐱_<i for the conditional distribution of each of the pixels is modelled using a deep convolutional neural network with residual connections. The convolution kernels are masked appropriately to ensure that the context of a pixel does not depend on the pixels after it. The conditional distribution is then modelled as a mixture of logistic distributions, where the parameters of the distribution depend on the context. This model is then learned on RGB images using maximum likelihood training.

Once the model is trained, it can be used to solve different inference tasks, as we describe below. Sampling from autoregressive models is slow because of their sequential nature which limits their utility. However, for our approach, we only require the gradients of the density p() with respect to the image X. This can be computed efficiently using backpropagation to the inputs.





Deep generative models and autoregressive model

Different generative models make different assumptions over the data to be modelled. Projection networks assume the graphical model to be 𝐳→𝐱. Where 𝐳 are the latent variables (noisy/blurred image in this case) and 𝐱 is some image. Since we are using CNNs, we are also assuming spatial redundancy. While in the case of autoregressive models, our graphical model is a directed 2D grid.  So, we are directly modelling the dependency between the pixels. 
This especially helps whenever the neighbouring pixels are highly correlated e.g. in the case of text, edges etc. One pixel provides a stronger signal to it’s neighbouring ones. 


By using CNNs, we assume spatial redundancy i.e. the similar pixels would be similar. Also, the dependency between the pixels in the final output is not directly modelled. Which is why it is generally tough to get sharp and smooth variations after deconvolutions. These two papers talk about these problems (https://arxiv.org/pdf/1702.06506.pdf, https://arxiv.org/pdf/1705.06820.pdf ) and how pixel level approaches have helped. Hence, modelling distribution over individual pixels should work better in producing sharp and smooth variations as compared to deconvolution based approaches. It should also be noted here that in pixelCNN, we are approximating things using CNN so there is spatial redundancy. But this spatial redundancy is between the distribution of the pixels and not the pixels themselves. It is assuming that the nearby pixels have similar conditional distribution which is a weaker assumption than assuming that assuming that nearby pixels would be similar as in case of auto-encoder based networks. 


Deep prior models used for solving inverse problems differ in the underlying assumptions they make for modelling densities. Chang et al. <cit.>, Sonderby et al. <cit.> and Bigdeli et al. <cit.> model the relation between the corrupted/noisy/blurred image and the clean image as a whole using deep convolutional networks. These networks are trained using adversarial and auto-encoder based losses to ensure that the output of the network is perceptually close to the clean image. However, there is no direct dependence between the pixels in the output image. This often leads to artifacts in the reconstructions with the value of a few pixels varying abruptly from those of the neighbouring pixels. These artifacts make the images look unnatural as natural images often have high correlation between adjacent pixels. Gao et al. <cit.> and Bansal et al. <cit.> also talk about the need of pixel-level deep approaches for the domain of dense classification. 

On the other hand, autoregressive models such as RIDE <cit.>, PixelRNN <cit.> and PixelCNN++ <cit.> model images as a 2D causal grid of pixels. This allows for tractable maximum likelihood estimation of densities. Moreover they explicitly handle the dependencies between pixels by modelling the conditional probability of a pixel given the pixels before it (in a raster scan). By utilizing such models as prior for ill-posed image reconstruction enables us to have better pixel-level consistencies in the reconstructed image. Hence, our reconstructions are better than the state-of-the-art in terms of perceptual quality and quantitative metrics such as PSNR and SSIM.








§ OPTIMIZATION METHODS FOR DEEP AUTOREGRESSIVE INFERENCE



In this section, we discuss inference methods for various forward models discussed earlier. We want the desired solution to have higher likelihood (lower NLL) under the image prior and at the same time satisfy the constraints specified by the forward model. For this, we perform projected gradient descent. We divide our approach into three categories based on the amount of noise and the kind of forward model. Hard constraint (equality) method is used when there is less or no measurement noise (Section <ref>). For certain imaging models like FlatCam, there is no closed form for the projection operator. We instead use the Augmented Lagrangian Method (ALM), see Section <ref>. For the cases of high noise, the measurements deviate significantly from the forward model, and the soft constraint method (inequality) is used (Section <ref>). Further, in Sections <ref> and <ref>, we describe two implementation hacks which have proved useful for our approach. 




 §.§ Hard constraint method


We first analyze the case when the measurement is directly obtained using the imaging model without any noise.  is then a deterministic function of  and hence the likelihood term would correspond to constraints. The problem can be formulated as 

    𝐗̂   = _ ( log(p())  such that  = f()

where f is provided by the imaging model. The signal prior model is the learned autoregressive model with parameters θ. Also, we constrain the intensity of the image to be between 0 and 1. Thus our problem is given by:

    𝐗̂   = _ ( log(p_θ())  s.t.  = f(), 0 ≤_ij≤ 1

Let 𝒞_1 and 𝒞_2 denote the constraint sets {: = f()} and {: 0 ≤_ij≤ 1   ∀ i,j } respectively.



We use projected gradient descent to solve this constrained optimization, which involves performing the following steps iteratively:

    𝐇_k    = _k + α∇_ log(p_θ(_k))
    𝐉_k    = Π_𝒞_1(𝐇_k)
    𝐗_k+1   = Π_𝒞_2(𝐉_k)

where Π_𝒞_1 and Π_𝒞_2 are projection operators to the constraint sets 𝒞_1 and 𝒞_2 respectively. For Eq. <ref> backpropagation to the inputs is used to get the data gradients. For Eq. <ref>, pixels in the image are clipped between 0 and 1 in every iteration. 

Π_𝒞_1 is different for different imaging problems. For the randomly missing pixels case, 

    𝐣_𝐤 = (1 - 𝐦) ∘𝐡_𝐤 + (𝐦) ∘𝐲

where 1 is an n^2 vector of ones. This implies that we should only be updating the missing pixels and leave the other pixels the same, which is intuitive. 

For Single Pixel Camera we have,

    𝐣_𝐤 = 𝐡_k-Φ^T(ΦΦ^T)^-1(Φ𝐡_k-𝐲)

where 𝐣_𝐤 and 𝐡_k are vector representations of matrices 𝐉_k and 𝐇_k respectively. We consider row-orthonormalized matrices for compressive sensing, hence ΦΦ^T is an identity matrix.

For LiSens case, similar to SPC, we have 

    𝐉_𝐤 = 𝐇_k-Φ^T(ΦΦ^T)^-1(Φ𝐇_k-).




 §.§ Augmented Lagrangian method



For the case of FlatCam reconstruction, the matrices Φ_𝐋Φ_𝐋^T and Φ_𝐑Φ_𝐑^T are ill-conditioned and can't be inverted. A closed form solution for projection operator doesn't exist. So, we consider the augmented Lagrangian corresponding to 𝒞_1, with a dual parameter λ.

    ℒ(,λ) = -log(p_θ()) + ρ - Φ_L Φ_R^T ^2_F 
     + ⟨λ,  - Φ_L Φ_R^T  ⟩_F

However, instead of minimizing the Lagrangian with respect to the primal variable in each iteration, we just take one step of gradient descent. We further separate the gradient descent into two steps, one entirely depends on the prior while the other entirely depends on the imaging model. The update steps are as follows.

    𝐇_k    = _k + α∇_log(p_θ(_k)
    𝐉_k    = 𝐇_k + Φ_L^T (λ_k -  ρ( - Φ_L _k Φ_R^T)) Φ_R 
    _k+1   = Π_𝒞_2(𝐉_k) 
    λ_k+1   = λ_k + ρ( - Φ_L _k Φ_R^T )




 §.§ Soft constraint method


Consider the case when the sensor has measurement noise,

    = f() + η

Assume the measurement noise η to be Gaussian distributed, i.e. 

    η   ∼𝒩(0,σ)
       ∼𝒩(f(),σ)

The MAP estimation problem can hence be reduced to 

    𝐗̂   = _ ( log(p_θ()) + λ - f() ^2 )

where λ has to be estimated if we do not know the standard deviation of the measurement noise. Since the constraints are not exact here, we replace the step to project to the constraint space by instead taking a step towards minimizing the likelihood. Hence, we replace Eq. <ref> by gradient descent over likelihood, 

    𝐉_k = 𝐇_k - α f'(𝐇_k) ( - f(𝐇_k))







 §.§ Stochastic gradients using pixel dropout


We observe that if we update all the pixels in the gradient update (Eq. <ref>), then we get washed out reconstructions. The autoregressive prior directly models correlation between neighbouring pixels. Hence it tends to assign same values to neighbouring problems. We combat this problem by randomly selecting a certain amount of pixels to update in each step. Hence, not all pixels get updates at every step. We call this pixel dropout, and for incorporating that, we replace the gradient in Eq. <ref> by stochastic gradients, i.e.,

    𝐇_k    = _k + α𝐌∘∇_ log(p_θ(_k))

where  𝐌 is a random binary mask with the percentage of zeros determined by the pixel dropout ratio. This is analogous to the case of training deep neural networks, where Stochastic Gradient Descent (SGD) helps in escaping from sharp local minima <cit.>. Here, the washed out reconstructions correspond to sharp local minima owing to the strong correlation between pixels. We demonstrate the effect of the amount of pixel dropout on the reconstructions in Section <ref>. 




 §.§ Splitting and Stitching


Our prior model is trained on 64 × 64 patches, hence the input for p_θ() has to be 64 × 64. While we perform the likelihood step on the entire image, our approach is designed such that the prior gradient update, projection, and clipping steps are separate. Before the prior gradient update, we split the image into a batch of 64 × 64 patches. Before performing the likelihood step, we stitch the patches back into original dimensions.  

Our approach is summarized as follows:



 






§ IMPLEMENTATION DETAILS




 §.§ Our Approach


We train PixelCNN++ on the downsampled 64×64 ImageNet data as introduced in <cit.> for 6 epochs. Batch size is kept as 36 and the number of filter channels as 100. The rest of the parameters are same as the ones used for training PixelCNN++ on 32×32 ImageNet in <cit.>. We obtain a negative log likelihood score of 3.66 on test data and 3.5 on train data which is consistent with the numbers reported in <cit.> for similar data. 

With this learned model, we use our proposed algorithm as described in Algorithm 1, for the experiments described below. An initial image is sampled from a uniform random distribution. However, we observe that starting with different initial images doesn't have much effect on the final converged reconstruction. We use momentum in the gradient update for faster convergence, with its value set to 0.9. Step size α, maximum iterations, likelihood weightage ρ for each experiment are mentioned in the subsequent section.

For reconstructing color images, we consider multiplexing along individual color channels. Hence, we have separate Φ matrices for all the three channels and obtain three separate measurement vectors  for each channel.

We have made the code of our implementation for the task Single Pixel Camera reconstruction available online[https://github.com/adaveiitm/deep-pixel-level-prior]. 



 §.§ One Network to solve them all


We use the original implementation of <cit.> available online[https://github.com/rick-chang/OneNet] with certain modification as mentioned below. 

For simulating color Single Pixel Camera, the original implementation rasterizes the entire N × N × 3 image into a single vector and creates one Φ matrix to compress this into a single measurement vector. We believe that this might not be feasible to implement in a real system. Hence, we modify their implementation to instead simulate separate Φ matrices for each channel as in Section <ref>.  

While simulating SPC measurements on large images, the original implementation only deals with local multiplexing. It breaks them down into patches of 64 × 64 and compresses each of these patches separately. We modify this to deal with the more challenging case of global multiplexing, where we compress the entire image. 

We extend the original implementation for LiSens and FlatCam as well, by considering the above modifications and incorporating the respective forward models. 

We use model provided which was trained on 64 × 64 Imagenet for 2 epochs for testing the results. We found that the results were very much dependent on the alpha parameter (penalty parameter) which had to be tuned for each image to get the best solution. 



 §.§ TVAL3

For comparisons with TVAL3 ( TV minimization by Augmented Lagrangian and ALternating direction algorithms ) <cit.>, we use the MATLAB implementation[http://www.caam.rice.edu/ optimization/L1/TVAL3/] with the default parameters. The number of iterations is set to 80. For color image reconstruction, we update each channel separately using TVAL3.




§ EXPERIMENTS


In this section, we present the reconstructions from our approach and compare them with the existing state-of-the-art approaches. To being with, we illustrate the ability of an autoregressive prior in reconstructing pixel level details using an example of missing pixel inpainting in an image. For this, we randomly mask out pixels from the image and use our prior to reconstruct these missing pixels. We perform by keeping the observed pixel values as same and update missing pixels to maximize the the prior loglikelihood. Specifically, we take an image of size 384x512 and mask 80% of the pixels in the initial image as could be seen in Figure <ref>. We compare our results with that of OneNet <cit.>, and we can observe details in our reconstruction much better like the text outlines, also quantitatively in terms of PSNR and SSIM. We use a step size of 75 and run for approximately 1000 iterations. 

For all the three imaging setups of SPC, Lisens and Flatcam we perform reconstructions on both simulated data and real measurements. In case of simulation we compare our reconstructions with TVAL3 <cit.> and OneNet <cit.>. In case of reconstructions from real measurements, we compare our results with TVAL3. OneNet experiments failed to converge to a stable point in this case hence we could not provide comparison with this approach. For real Lisens at 66% measurements, although OneNet converges, results obtained were very poor compared to other approaches.




 §.§ Single Pixel Camera




  §.§.§ Simulation case

We show quantitative and qualitative comparisons of simulated SPC reconstruction results on images of sizes 128×128 and 256×256 respectively as shown in Table <ref> and Figure <ref> respectively. Measurement rates considered are 10% and 25% for 128×128 and 5% and 10% for 256×256. Similar to RIDE-CS <cit.>, we generate the ϕ matrix as a random Gaussian with orthonormal rows. We perform gradient descent and projection operation on the compressed image for 2000 iterations in the case of 25% measurement rate and for 2500 iterations in case of 10% measurement rate. We use a step-size of 7.5 and the hard constraint projection method. In all cases, we intialize with random image from uniform distribution.  We compare our results to <cit.> and we are able to show significant improvement in reconstruction results in terms of PSNR and SSIM values. Our reconstructions have better edges and textures compared to the reconstructions from OneNet.






  §.§.§ Real Case








We show our real SPC reconstruction results in Figure <ref>. Data for this experiment is provided to us by the authors of <cit.>. We obtain the real SPC sensor measurements at 30% and 15% measurement rate respectively. The images we reconstruct in this case are grey scale images. Here also, we use the Hard constraint projection method for inference. We compare our results with TVAL3 and RIDE-CS <cit.>. Our method performs better than both RIDE-CS and TVAL3 in terms of PSNR and SSIM values. Apart from these measures, we observe that our method produces a sharper reconstruction. We use the same hyperparameters and training procedures as in the simulated case. 



 §.§ LiSens




  §.§.§ Simulation case

 The reconstruction in case of simulated  LiSens is done at 25% and 40% measurement rates. Our LiSens experiments, similar to SPC experiments, have been done on both 128x128 and 256x256 images as shown in Table <ref> and Figure <ref> respectively. We compare our reconstructions with that obtained using OneNet. Our method provides better results in terms of visual perception as well as PSNR and SSIM values. Our reconstructions have well-defined boundaries of different objects in the image and do not produce artifacts which are observed in case of OneNet. We have used hard constraint case for the simulated LiSens reconstruction for approximately 2000 iterations with a step-size of 7.5.






  §.§.§ Real measurements


 The real LiSens experiments have been done at 16% and 33% measurement rates obtained at a resolution of 768×256 , as provided by the authors of <cit.>. We compare our real Lisens with TVAL3 as in Figure <ref>. Our method performs better reconstruction with respect to low level details in the image. Our proposed method’s reconstruction has little or no blur compared to TVAL3 and the reconstruction is sharper in terms of object boundaries in the image. We use Hard constraint method for reconstruction with 25% dropout in pixel-wise update. We use an update step of 7.5 and 2000 iterations for reconstruction, similar to simulated experiment. 



 §.§ FlatCam







  §.§.§ Simulation case

The matrices Φ_L and Φ_R in the FlatCam imaging model are estimated based on the calibration procedure mentioned in <cit.>. As we want to deal with RGB images, separate Φ_L and Φ_R matrices are calibrated for each of the R, G and B channels with the help of a Bayer color filter array on the sensor. We compare our results with OneNet and L2 regularisation, on two 256x256 images as shown in Figure <ref>. Our method shows better PSNR, SSIM, and perceptually better quality samples. Our method produces the least blurry solution and objects in the image has well defined boundaries. We use 25% pixel dropout and perform 1000 iterations of augmented Lagrangian method with the step size α as 60.0 and ρ as 10. 






  §.§.§ Real measurements


We use the data provided by the authors of <cit.>. The original images were displayed on a monitor and captured using FlatCam. Using a Bayer color filter on the sensor, separate measurements for the three color channels can be obtained. We compare our reconstructions with L2 regularization as shown in Figure <ref>. Our reconstructions are more accurate in terms of brightness, boundaries and sharpness of the image. We use soft constraint case for reconstruction and use the same hyperparameters as in the simulation case. 



We observe that reconstructions from real FlatCam are not qualitatively as good as with real SPC and LiSens measurements. This is because the forward model assumed in this case is erroneous. Firstly, there are calibration errors in estimating the Φ_L and Φ_R matrices. Secondly, the forward model in <cit.> relies on the separability assumption leading to model error. 



 §.§ Ablation Experiments




  §.§.§ Effect of pixel-wise dropout


In this experiment, we vary the amount of pixels not updated in each iteration and observe its effect on the reconstructed image, see Figure <ref>. When the dropout ratio is zero, the area in the image having texture is over smooth. With considerable dropout ratio (25%), the texture is reconstructed better amounting to a higher PSNR and SSIM. However, on increasing it further, the reconstructions appear noisy with a reduction in quality. Thus, for all our experiments, we used 25% dropout.   






  §.§.§ Comparison with Ride-CS - grayscale SPC

While we train our model on colored Imagenet data, we observe that in practice this approach works well on reconstructing grayscale images as well. We compare our reconstruction with that of RIDE-CS <cit.>, which uses the autoregressive model RIDE <cit.> as image prior. In Figure <ref>, we compare the reconstruction of a grayscale image from Single Pixel Camera measurements using our approach and RIDE-CS for 15% measurement rate. The reconstruction obtained from our approach is better than that of RIDE-CS. This is because we use PixelCNN++ which is a deeper network than RIDE and hence has better representation power.  Also, the running time of our approach (∼ 5 minutes ) is much less than that of RIDE-CS (∼ 30 minutes). Our approach is CNN based and hence can be parallelized over multiple GPUs while RIDE-CS relies on a network of spatial LSTMs which are tough to parallelize. 








  §.§.§ Comparison with OneNet in their original setting

Till now we have performed all the experiments with different Φ matrix for each color channel. However in OneNet <cit.>, the authors have considered one Φ matrix that multiplexes across the three color channels, which might not be feasible to implement in a real system. For this ablation experiment, we consider the original setting as used in <cit.> and compare their reconstructions with ours for 10% SPC reconstruction on the 9 test ImageNet images mentioned in the <cit.>. PSNR and SSIM values for the same are mentioned in Table <ref>. Our approach performs better than OneNet. 










  §.§.§ Inference in the presence of noise


We analyze the performance of the three proposed inference methods (hard constraint, soft constraint, and ALM) for the case of SPC and LiSens. In case of FlatCam, we compare only the performance of soft constraint-based inference and ALM. Hard constraint-based inference cannot be used in Flatcam as the matrix ΦΦ^T is non-invertible. 
For noise analysis of the inference algorithms, we add independent and identically distributed Gaussian noise at each measurement. In all the simulations, input image is normalized to [0,1]. We consider the cases of no noise and noise with σ values of 0.001, 0.01 and 0.05. For SPC, we consider a measurement rate of 10% and for LiSens 25% measurement rate. The image used for these experiments is the 'parrot' image.

The results are shown in Figure <ref>. For SPC and LiSens, the hard constraint case performs best when noise is less. As expected, the performance of the hard constraint case degrades as the noise increases. Overall, soft constraint-based method works well at all noise levels. However, it has an additional parameter λ, which needs to be set appropriately based on the noise level. On the other hand, there is no such parameter for the hard constraint case, and hence it is preferable, especially for the low noise case. In fact, for our real experiments on SPC (Figure <ref>) and Lisens (Figure <ref>), we use hard constraint-based inference, which produces reasonable results. For the Flatcam real experiments, we use soft constraint-based optimation as ALM also has an additional parameter ρ which needs to be set according to the noise level.



















§ DISCUSSION AND CONCLUSION

We demonstrate the efficacy of deep pixel level image prior for ill-posed reconstruction in different computational imaging problems. Among the three proposed approaches for inference, hard and soft constraint based and ALM based, overall, soft constraint-based method works well and can handle noisy measurements by appropriately varying the tuning parameter, λ. However, when there is no noise or less noise in the measurements, the hard constraint-based method performs as good as soft constraint case with an additional advantage of being parameter free and hence is preferable. In fact, for our real experiments on SPC (Figure <ref>) and Lisens (Figure <ref>), we use hard constraint-based inference, which produces reasonable results. For cases such as Flatcam, non-invertibility of ΦΦ^T prevents the use of hard-constraint based inference.

Our approach enjoys the versatility of image priors and rich feature representation of deep neural networks. Being pixel level, it explicitly accounts for pixel level correlations resulting in consistent texture and edges. We show our evaluations on both the simulation of forward models and data from real setups. In all cases, both quantitative and qualitative metrics suggest that our approach performs better than traditional methods and current state-of-the-art learning based methods. An interesting line of work would be to incorporate deviations from the forward model, due to calibration and model errors, in our approach to further improve the quality of reconstruction for FlatCam. 




§ ACKNOWLEDGMENT

This work is supported by Qualcomm Innovation Fellowship (QInF) 2016 and 2017. We would like to thank Dr. Aswin Sankaranarayanan and Jian Wang from CMU  for sharing the real measurements for SPC and LiSens setup. We would like to thank Dr. Ashok Veeraraghavan, Vivek Boominathan, Jasper Tan from Rice University for sharing the FlatCam data and for useful discussions. 




































IEEEbib1
























Michael Shell
Biography text here.



John Doe
Biography text here.






Jane Doe
Biography text here.























