Abstract
This paper presents a video encoding method in which noise is encoded using a novel parametric model representing spectral envelope and spatial distribution of energy. The proposed method has been experimentally assessed using video test sequences in a practical setup consisting of a simple, real-time noise reduction technique and High Efficiency Video Codec (HEVC). The attained results show that the use of the proposed parametric modeling of noise can improve the subjective quality of reconstructed video by approximately 1.8 Mean Opinion Scope (MOS) points (in 11-point scale) related to the classical video coding. Moreover, the present work confirms results attained in the previous works that the usage of even sole noise reduction prior to the encoding provides quality increase.
Introduction
In recent years, the overall share of video transmission in total internet traffic was increasing systematically. Some sources estimate [1] that video broadcasters such as YouTube, Netflix, and the VoD (Video on Demand) services already attract approximately one-third of all internet users. It was estimated [2] that video accounted for 73% of total internet traffic in 2016 and it will likely increase to 82% by 2021. A vast number of subscribers, approximately 125 million [3] , force service providers to search for new, highly efficient technical solutions. This need stimulates both the development of internet infrastructure and development of more efficient video compression technologies.
Introduction
During three decades of research on video coding, many techniques have been developed in academia, research institutes and industry [4] . The most influential incubators of successive generations of video compression techniques are Motion Picture Experts Group (MPEG) working on behalf of International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC), and VCEG (Video Coding Expert Group) who work on behalf of International Telecommunication Union (ITU). The cooperation of these two bodies, e.g. under the form of Joint Collaborative Team on Video Coding (JCT-VC), resulted in the following development of technologies adopted as international standards: MPEG-2 [5] , Advanced Video Coding (AVC) [6] [7] , High Efficiency Video Coding (HEVC) [8] [9] . These standards correspond to successive generations of compression technology, each of which provides approximately 50% reduction of bitrate required for the same quality of decoded video with respect to the previous one (e.g. HEVC with respect to AVC) [4] [7] . Such gain is achieved through the use of increasingly complex compression algorithms. Currently, MPEG and VCEG are jointly working on Versatile Video Coding (VVC), which is expected to be standardized in 2020 [10] . Additionally, industrial consortia have also proposed several complete compression technologies (e.g. recent AV1 codec [11] ). Natural video content, captured with real cameras, is inherently characterized by the presence of noise originating from photosensitive electrical circuits, often referred to as film grain. Although the quality of optical sensors improves over time, which theoretically allows for higher Signal-to-Noise Ratios (SNRs), the same occurs in the context of camera resolutions and frame rates. Higher resolutions lead to smaller pixels on sensors, while higher frame rates enforce shorter exposure times. As a result of both of these factors, the amount of noise present in video often remains significant, although the exact amount and characteristics of noise varies [12] . Even with modern camera technology, noise remains noticeable, especially under poor illumination or for low-end consumer cameras [13] .
Introduction
Noise in video disturbs many processes of video compression, such as inter prediction, intra prediction, mode selection, and quantization. Since noise is random by nature, it cannot be efficiently predicted, and its existence reduces compression performance. In practice, this implies that an encoder would spend bits for an inefficient representation of noise. Therefore, a common solution involves removal or reduction of noise prior to encoding (see [14] , [15] , [16] ). Although the advantage of such an approach is that the bitrate is significantly reduced, information regarding noise is not scottjames@example.com. This represents a disadvantage, since noise is perceptually important to viewers [17] as the absence of noise causes a sense of unnaturalness. This paper addresses the mentioned aforementioned issue of compression of video with noise. Instead of simple noise removal, the paper proposes to estimate the noise parameters in the encoder and transmit these parameters in the encoded bitstream. This is a similar approach to the one already successfully used in audio coding, e.g. in the Perceptual Noise Substitution (PNS) tool in Advanced Audio Codec [18] . A similar idea in the context of video has been employed in previous works [19] , [20] and [21] , where authors also proposed the use of parametric noise model and noise synthesis on the decoder side. Notably, works [19] and [20] were performed in the context of older AVC video coding technology. These works provided very limited experimental verification since only a few sequences were used. Moreover, in [20] , only objective results are provided, while [19] provided only very limited subjective quality results. Work [21] is performed in the context of the AV1 video codec and it also considers parametric noise modeling. In particular, a two-dimensional autoregressive model is applied for modeling of the spectrum and the amplitude of the synthetized noise is modeled with piecewise linear function. Also in this work a very limited set of experimental results is provided (based on four sequences) with no formal subjective evaluation performed.
Introduction
Notably, none of the aforementioned works assess whether the attained gains originate solely from noise reduction or from the addition of synthetic noise. These issues are considered in the present paper.
Introduction
Therefore, the significant contributions of this paper include: -Proposal of a novel parametric model of noise, consisting of two components: Spatial Distribution of Energy (low-resolution video) and Spectral Envelope (encoded with Log-Area-Ratio coefficients [22] ).
Introduction
-Implementation of the abovementioned idea on top of the state-of-the-art HEVC video coding technology.
Introduction
-Exhaustive experimental results of subjective evaluation, providing a comparison of the performance of the proposed method versus classical video coding scheme -with and without noise reduction.
Proposed coding scheme
The general idea of the proposal (Fig. 1) consists in coding of video with the use of two layers: -The base layer, which contains useful content that can be efficiently modeled with existing predictive tools.
Proposed coding scheme
-The noise layer, which contains unpredictable random content that can be modeled statistically.
Proposed coding scheme
While the base layer is coded using standard methods, only a parametric model is transmitted for the noise layer. At the decoder, the parametric model is used to synthesize noise, which resembles the original one, inputted to the encoder.
Encoder
The first step of the processing on the encoder side ( Fig. 1) The base layer video ( ) is encoded with a general-purpose video encoder (Fig. 1) , such as AVC, HEVC or AV1. Due to the decomposition, the base layer contains significantly reduced amount of noise and can thus be efficiently modeled using predictive compression tools. In particular, the potential benefits of such an approach include:
Encoder
-Intra and inter prediction may be more efficient, thus making the prediction error potentially smaller,
Encoder
-Motion vectors for inter-frame prediction may be more consistent and represent the real motion in video, thus being more susceptible to compression.
Encoder
-The structure of the compressed image and selection of encoding modes may be more regular. Therefore, control elements (e.g. signaling of CU hierarchy partitioning in HEVC) may be compressed more efficiently in the course of entropy encoding. The noise layer ( ), on the other hand, contains unpredictable random content. Encoding of such signals with generalpurpose video coding techniques is very inefficient. Therefore, the present paper proposes parametric modeling the noise layer.
Encoder
For this purpose, two components that describe spatial and spectral characteristics of the modeled noise are proposed:
Encoder
-Spatial Distribution of Energy (SDE), and -Spectral Envelope (SE).
Encoder
Whole noise modeling is performed independently for each color channel (e.g. Y, C b ,C r ). As such, for the sake of brevity, the following explanation is provided in the context of a single channel. (1):
Encoder
where:
Encoder
( ) -value in the noise layer video at pixel coordinates ( ) in the current frame,
Encoder
-width and height (in pixels) of the processed video, Notably, the resolution of SDE ( × ) is smaller than the resolution of the input video ( × ) depending on the size of blocks used. Therefore the bitstream required to represent SDE is negligible (about 1% depending on the case -see: Table 7 compression [22] . First, the values in the noise layer video ( ) are normalized with respect to RMS values that were calculated beforehand and stored in SDE (2):
Encoder
This step is required in order to attain stability of the LPC algorithm, which would be otherwise unstable due to energy fluctuations in the signal [22] .
Encoder
Next, the linear predictive model is applied in the horizontal and vertical direction in order to estimate IIR filters parameters. This is attained by the construction of one-dimensional signals of concatenated rows and columns of ( ), respectively:
Encoder
Therefore, normalized noise layer video ( ) is rearranged into a single row signal ( ) and into a single column signal ( ). For each of these, the classical linear predictive model of order is used, which can be expressed by the general approximation equation (4) for a certain signal ( ):
Encoder
The linear prediction weights … are coefficients of an IIR filter (6) that represent the spectrum of the analyzed signal ( ).
( ) ∑ ( )
Linear prediction is performed for both the horizontal and vertical direction, so that ( ) ( ) and ( ) ( ), which yields IIR filter transfer functions ( ) and ( ). This is performed individually for each frame of the noise layer video.
( ) ∑ ( )
The proposed statistical model of noise can be summarized to consist of:
( ) ∑ ( )
-An image of the Spatial Distribution of Energy ( ),
( ) ∑ ( )
-Spectral Envelope (SE) expressed by the transfer functions of filters ( ) and ( ) , both for each frame of the noise layer. The exemplary amounts of data required for transmission of the proposed parametric model of the noise are presented in Table 7 (in experimental results section), e.g. for HD sequences (class B1) it is required to transmit about 4 kbit/s of SE data and 4 kbit/s for SDE data.
Decoder
On the decoder side ( Fig. 1) , the noise layer is synthesized using the coded parameters of the noise (Fig. 2) .
Decoder
First, white Gaussian noise video ̃ ( ) is generated. Resolution of this video is the same as that of the input video ( × ). Next, the generated noise is filtered in the horizontal and in the vertical direction using IIR filters with transfer functions ( ) and ( ), respectively. The result is noise video ̃ ( ), which is spectrally shaped to resemble the spectral properties of the original noise layer ( ). Next, the values from ̃ ( ) are scaled (multiplied) with the corresponding values from the Spatial Distribution of Energy ̃( ), which is also reconstructed at the decoder side. This provides the reconstructed noise layer ̃( ), which resembles both the spectral and spatial properties of the original noise layer ( ). Finally, the synthesized noise layer ̃( ) is added ( Fig.1) to the reconstructed base layer ̃( ) to produce the output reconstructed video ̃( ).
Implementation
For the sake of experimental verification, the general idea of the proposal described in Section II is further elaborated and implemented as described below. The whole processing was performed in YC b C r color space.
Implementation
Noise reduction (Fig. 1) was realized with the use of the motion-compensated technique implemented in "mv-tools" motion compensation software package [23] , which is a plug-in for the VirtualDub/AviSynth video scripting framework [24] . It consists of finite impulse response (FIR) filtering along the direction of movement in the video, with weights adaptively selected depending on the content.
Implementation
The input frames neighboring in time (next and previous ones) are used to reduce noise in the currently processed frame, denoted as ( Fig. 3) . At first, each neighboring frame, denoted as , * + is used to predict the current frame with the use of block-based motion compensation, thereby yielding . This step is similar to the motion vector search known from video coding. Next, the predicted images are averaged together with the current frame , with weights depending on the similarity of particular predicted image to the current frame . A variant of the technique with six neighboring frames (3 next and 3 previous ones, thus ) was used, due to its good performance versus computational complexity (Fig. 3) . More details can be found in [23] .
Implementation
For the role of General-Purpose Video Encoder and Decoder (Fig. 1) , HEVC video coding technology was selected along with its implementation in MPEG HEVC model software HM 13.0. The coding configuration recommended by JCT-VC in "Common
Implementation
Test Conditions" [25] was used, e.g. Intra period of 32 frames. For the compression of SDE, HEVC was also selected. Uniformly (for all bitrates and sequences), QP value of 30 was used.
Implementation
Notably, such a small resolution of image (as 64×36 pixels) fits into a single HEVC Coding Unit (CU), thus generating a very small compressed bitstream.
Implementation
Spectral Envelope (SE) was estimated using the linear prediction of order . IIR filter parameters were estimated with the use of the Levinson-Durbin recursion algorithm [26] . For the sake of compression, PARCOR coefficients were used instead of classical filter coefficients . coefficients, also called reflection coefficients, which are the by-product of the Levinson-Durbin algorithm are known to be much less vulnerable for lossy compression than coefficients. A known solution based on log-area-ratio (LAR) [22] representation was used (6):
Implementation
All of the LAR coefficients were coded using 8-bit representation.
Motioncompensated prediction
Frame f-2
Motioncompensated prediction
Frame f-1
Motioncompensated prediction
Frame f-3
Motioncompensated prediction

Frame f+2
Motioncompensated prediction
Frame f+3
Motioncompensated prediction 
Experimental results
The main idea of the proposal consists in encoding noise parameters, and using them in the decoder to synthesize noise, which resembles but is not identical to that on the input. Therefore, the estimation of such popular objective quality parameters such as Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), or Structured Similarity Index (SSIM) is irrelevant [27] . Thus, the methodology used during the present experiments was based on subjective quality assessment. In particular, it was based on tests performed for the evaluation of results of "Call for Proposals" [28] issued by MPEG in search for video compression tools for HEVC. In particular, the same viewing method (DSIS -Double Stimulus Impairment Scale) [30] , the same sequences (Table 1) , and the same bitrates (Table 2 ) [29] were used in this paper. Please note that an additional bitrate Rate 0 (lower than Rate 1) has been included in addition to [28] in order to allow better HEVC-based codec comparison. In total, experiments used 19 sequences divided into 6 classes with various resolutions and frame-rates. During the assessment performed for the sake of the present work the Double Stimulus Impairment Scale (DSIS) method in accordance with ITU Recommendation BT.500 [30] was used. Viewers interchangeably viewed the reference (uncompressed original video) and tested case (video encoded and decoded with the tested codec), which was repeated twice for each case before the voting. As per the evaluation performed in work [29] , a particular variation Mean Opinion Score (MOS) was applied, using quality scale from 0 to 10, instead of the classical "impairment" quality scale. The tested codecs were configured to fit the encoded sequences within the required bitrates (Table 2) , which were identical for all codecs. As such it was possible to directly compare the quality since the compared cases shared the same bitrate. The compared codecs were as follows:
Experimental results
A. Anchor -original video (with noise) encoded with standard HEVC.
Experimental results
B. Noise reduction without synthesis -video is denoised prior to the coding but the noise is not synthesized at the decoder.
Experimental results
C. Proposal -video encoded according to the proposal (as base and noise layer) including the addition of synthetic noise to the reconstructed video at the decoder side.
Experimental results
The idea behind including case B was to determine whether gains of the proposal originated from the denoising of the input sequence or from the addition of the synthetic noise in the decoder. In fact, since the bitstream overhead of the proposed noise parameters is negligible, it can be noted that case B is equivalent to the proposal (C) but without adding of the synthetic noise in the decoder. Although there is potential gain (of about 1% depending on the case -see: Table 7 ) from not sending noise parameters, it would not allow to change the quality of reconstructed video, e.g. to increase quality of the base video layer by selecting smaller quantization parameter (QP) ( Table 3) . The subjective evaluation tests were performed in many, 20-minute sessions, performed in different time, in order not to overstress the visual systems of viewers. In total, there were 94 subjects. Due to the high number of subjects the attained 95%
Experimental results
confidence intervals are very small -to of order of ±(0.06÷0.18). Confidence intervals have been marked on all graphs presenting the results (Figs. 4-9 ).
Experimental results
The example frames of the sequences are presented in Fig. 10 . For the sake of comparison, results for the middle bitrate (R3) are shown. Notably, the noise synthesized at the decoder with the proposed method (Fig. 10e) is practically indistinguishable from the original noise (Fig. 10a) . Moreover it can be seen that Anchor encoding (Fig. 10c) was not able to represent the original noise in the compressed bitstream and also wasted some bits trying to do so, which, due to limited total bitrate available, deteriorated the quality of the video itself. S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19
Experimental results
A. Anchor B. Noise reduction C. Proposal S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19
Experimental results
A. Anchor B. Noise reduction C. Proposal S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19
Experimental results
A. Anchor B. Noise reduction C. Proposal Based on the results it is evident that -in practically all of cases -the usage of noise reduction prior to compression (B)
Experimental results
statistically significantly increases the quality of compressed video versus the anchors (A). Except of the negligible cases of sequences S12 and S18 for a single rate-point Rate 0 (Fig. 4) , none of the confidence intervals for (B) and (A) overlap. The observed gain is approximately 0.5 to 2.0 MOS points. This confirms the results attained in previous works [15] and [16] . It can be noticed that the highest bitrate (R5) has been selected too high, since the scores provided by the subjects were saturated at about 10 (maximum), making the observation of differences between the analyzed cases more difficult. This rate has been used in the work [28] in order to allow comparison of AVC versus HEVC and is impractical in HEVC versus HEVC comparisons.
Experimental results
Moreover, the usage of the proposed parametric modeling of noise (C) increases quality gain even further, to range of about 1.0 to 2.0 MOS points. This difference is statistically significant in almost all cases with relation to the anchors (A), again with exception of sequences S12 and S18 for Rate 0. In relation to the noise reduction (B), the increase of quality of the proposed case (C) is about 0.2 to 0.5 MOS. In most of the cases, the quality gain is statistically significant, especially for lower bitrates. For example, the proposal (C) is statistically significantly better than (B) in the case of 15 sequences out of 19 sequences in total, when Rate 0 is considered (Fig. 4) and the case of 16 sequences out of 19 sequences in total, when Rate 1 is considered (Fig. 5 ). S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19
Experimental results
A. Anchor B. Noise reduction C. Proposal 7 8 9 10 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19
Experimental results
A. Anchor B. Noise reduction C. Proposal
Experimental results
This results is slightly worst when higher bitrates are considered -e.g. when Rate 4 is considered, the proposal (C) is better than (B) in the case of only 4 sequences. In other cases, the confidence intervals overlap, but also it can be noted, that the proposed usage of the parametric modeling of noise (C) is never worse than (B) or (A).
Experimental results
The average results of the evaluation are provided in Tables 4 and 5 The Table 7 concludes the amount of data required for transmission of the proposed parametric model of the noise used in the experiments. For Spatial Density of Energy (SDE) constant QP=30 value has been used in HEVC encoding. In the case of Spectral Envelope (SE), as mentioned prediction of order was used for both horizontal and vertical IIR filter parameters.
Experimental results
This resulted in total 20 coefficients, each represented 8-bit LAR coefficients. Thus, for each frame, transmitted were 160 bits.
Experimental results
Depending on the frame rate of particular sequences, this gives the bitrate required for representation of SE. Therefore depending on the rate-point and sequence (Table 1) , the bitrate required for transmission of the proposed parametric model is about 0.2-5% of the total bitrate ( Table 2) . As it can be seen in Fig. 10 , this is sufficient to represent a noise which is visually resembling the original noise. Also objectively (Fig. 11) , the spectrum approximated with LPC method is in agreement with the spectrum of the original noise. The bitrate reductions that can be attained with the use of the proposed method are not discussed in the present paper due to the following reasons:
Experimental results
-It is irrelevant to use Bjøntegaard [31] method based on PSNR, due to the addition of a synthetic noise to the decoded video, proposed in the present paper.
Experimental results
-The bitrate reduction originating from the use of sole noise reduction (case B), in a very similar coding scenario, has already been analyzed in the previous works [15] , [16] and are reported to be about 50% on average.
Experimental results
-The bitrate of the proposed method (C) is practically the same as of sole noise reduction (B), because the bitstream related to the proposed noise model is negligible, of about 1%, depending on the case (see : Table 7 ), of the total bitrate. 
Conclusions
A video encoding method in which noise is encoded using a novel parametric model has been proposed, and its implementation has been deliberated. The proposed model of noise consists of two components: Spatial Distribution of Energy (low-resolution video) and Spectral Envelope (encoded with LAR coefficients).
Conclusions
Although the proposal has been presented in the context of HEVC video coding technology, its idea is general and can be applied in conjunction with any general-purpose video codec.
Conclusions
Exhaustive subjective tests were performed to assess the performance of the proposed technique and to compare it with classical video coding and simpler methods which employ only noise reduction without noise modeling. The experiments confirm some results already attained in other works demonstrating that applying noise reduction prior to video compression result in considerable video quality increases of approximately 1.44 MOS points. The results also highlighted that the proposed model can be used to increase the quality of compressed video even further -by approximately 0.3-0.4 Mean Opinion Score (MOS) points. In total, the gain of the proposed method is about 1.71 MOS points, as compared to the classical video coding without any noise-related processing. Finally, it has been shown that use of the proposal is justified due to its low computational complexity.
Conclusions
Apart from interesting experimental results for this particular proposed and implemented technique, the very important outcome of the paper is evidence that the use of synthetic noise added to the video reconstructed at the decoder side can increase its subjective quality.
Conclusions
Avenues for the future research include using more advanced noise reduction techniques and investigating the proposed method in the context of different video codecs: monoscopic (e.g. AV1) or multiview (e.g. HEVC-3D). 
|F(jω)|

Normalized frequency ω
Original Noise Spectrum LPC filter

{'paper_id': '135465006', 'title': 'Video coding technique with parametric modeling of noise', 's2_url': 'https://api.semanticscholar.org/CorpusID:135465006', 'source': 'S2ORC'}