ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images
Weiqi Li Shijie Zhao Bin Chen Xinhua Cheng Junlin Li Li Zhang Jian Zhang

















liweiqi@stu.pku.edu.cn








With the advent of virtual reality technology, omnidirectional image (ODI) rescaling techniques are increasingly embraced for reducing transmitted and stored file sizes while preserving high image quality. Despite this progress, current ODI rescaling methods predominantly focus on enhancing the quality of images in equirectangular projection (ERP) format, which overlooks the fact that the content viewed on head mounted displays (HMDs) is actually a rendered viewport instead of an ERP image. In this work, we emphasize that focusing solely on ERP quality results in inferior viewport visual experiences for users. Thus, we propose ResVR, which is the first comprehensive framework for the joint Rescaling and Viewport Rendering of ODIs. ResVR allows obtaining LR ERP images for transmission while rendering high-quality viewports for users to watch on HMDs. In our ResVR, a novel discrete pixel sampling strategy is developed to tackle the complex mapping between the viewport and ERP, enabling end-to-end training of ResVR pipeline. Furthermore, a spherical pixel shape representation technique is innovatively derived from spherical differentiation to significantly improve the visual quality of rendered viewports. Extensive experiments demonstrate that our ResVR outperforms existing methods in viewport rendering tasks across different fields of view, resolutions, and view directions while keeping a low transmission overhead[The complete code and pre-trained models of our method will be made available.]^* Interns in MMLab, ByteDance.^† Corresponding author..







<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224</concept_id>
       <concept_desc>Computing methodologies Computer vision</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Computer vision









  

  
    < g r a p h i c s >

  

  Proposed ResVR compared to previous ODI rescaling and viewport rendering paradigms. (a) Conventional methods focus on improving the quality of rescaled ERP images, resulting in inferior visual experiences. (b) Considering the fact that the desired content viewed on HMDs is a rendered viewport instead of an ERP image, our ResVR directly optimizes the quality of the final viewport for users through a novel discrete pixel sampling strategy and a spherical pixel shape representation technique. (c) Visual and PSNR comparisons of rendered viewports between the two pipelines in (a) and (b) on user HMDs.
  
  









ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images
    Weiqi Li^*1,2, Shijie Zhao^† 2, Bin Chen^1, Xinhua Cheng^1, Junlin Li^2, Li Zhang^2, Jian Zhang^† 1

^1Peking University         ^2ByteDance Inc
    April 29, 2024
====================================================================================================================================================







§ INTRODUCTION

With the growing interest in virtual reality and augmented reality, omnidirectional images (ODIs), also referred to as 360° or panoramic images, attract great attention within the computer vision community for their immersive and interactive capabilities. Although ODIs can capture scenes across a comprehensive 360°×180° views, head-mounted displays (HMDs) often present a limited field-of-view (FoV), necessitating resolutions as high as 4K×8K <cit.> to preserve details in a small viewport. High-resolution (HR) ODIs are typically stored on cloud servers by platforms of virtual reality media, requiring real-time download by users. This can degrade the visual experience of users, particularly under poor internet conditions.


Image rescaling <cit.> emerges as an effective method to reduce image file size for storage and transmission while preserving high quality in the reconstructed images at the user end. This technique first downscales HR images to low-resolution (LR) ones that keep the most important visual details, then upscales them back to their original HR versions. It not only minimizes the file size of LR images but also maintains the quality of reconstructed HR images <cit.>, thus becoming a straightforward approach for efficient transmission and storage for ODIs from cloud servers to user HMDs. However, the prevalent storage and transmission format for ODIs, i.e., the equirectangular projection (ERP), has directed state-of-the-art ODI rescaling works <cit.> to focus on enhancing the quality of ERP images. As depicted in Fig. <ref> (a), after receiving an LR ODI, the typical process on HMDs is two-step, involving (1) upscaling HR ODIs in ERP format from LR images (↦), and (2) projecting them onto the viewport (↦) using traditional interpolation methods such as bilinear. This pipeline does not fully account for the ultimate rendered image of the HMD viewport , particularly lacking optimization for the final viewing experience. As Fig. <ref> (c) shows, the quality of images seen by the user on HMDs can be significantly lower than anticipated.


In this paper, we point out that (1) the content viewed on HMDs is actually a rendered viewport, not an ERP image, and (2) focusing solely on the quality of ERP images will result in sub-optimal viewport visual experiences. To improve the viewing experience for users, there is a need to develop a comprehensive solution that is optimized for end-to-end ODI processing from the storage of ERP images to the display of the final viewport. To this end, we propose ResVR, a novel framework for joint Rescaling and Viewport Rendering of ODIs, marking an innovative step towards comprehensive end-to-end ODI processing. As shown in Fig. <ref> (b), ResVR aligns the optimization of network parameters with our primary goal of improving the quality of the final viewport. By utilizing such a new methodology, the viewports rendered through our ResVR framework exhibit enhanced details and fewer artifacts compared to those produced by conventional pipelines, as shown in Fig. <ref> (c).

In our proposed ResVR, HR ODIs are firstly embedded into LR images to facilitate efficient transmission. Then, HR viewports are directly rendered from LR ERP images on HMDs. This process (↦) does not need to produce HR ERP images. To deal with the irregular correspondence between the ERP area and the viewport, which hinders the joint optimization of downscaling and viewport rendering using traditional image-level loss training methods, we develop a discrete pixel sampling strategy. In each training iteration, this strategy is used to randomly sample paired sets of ground truth pixels and the reconstructed ones on viewports, thus making the end-to-end learning of our entire ResVR pipeline feasible in implementation. Furthermore, to enhance our ResVR's awareness of the positions of different viewport pixels on the spherical surface, we introduce a technique for spherical pixel shape representation. This technique employs spherical differentiation to calculate the geometric orientation and curvature of various viewport areas, offering positional information that contains more precise spherical attributes than existing 2D image representation methods <cit.>. This advanced representation effectively improves the quality of the final viewport, especially in regions of high latitude and longitude. Extensive experiments on various panoramic image datasets show that ResVR outperforms existing methods in multiple viewport rendering tasks across different FoVs, view directions, and resolutions. In summary, our contributions are: 


113 We propose ResVR, a novel framework for the comprehensive processing of omnidirectional images, seamlessly integrates image Rescaling with Viewport Rendering. Our ResVR effectively balances the transmission efficiency and users' visual experience.

113 We develop a discrete pixel sampling strategy to tackle the complex correspondence between viewport and equirectangular projection (ERP) areas within our framework. This strategy makes the end-to-end training of our whole processing pipeline feasible. 

113 We introduce a spherical pixel shape representation technique based on spherical differentiation to guide viewport rendering, which significantly enhances the visual quality of the final viewport.

113 Extensive empirical evaluations on various panoramic image datasets exhibit that ResVR consistently achieves new state-of-the-art visual quality while maintaining a low transmission bitrate.







§ RELATED WORK



 §.§ Omnidirectional Image Super-Resolution

Image super-resolution (SR) seeks to construct HR images from LR ones. Since the advent of deep neural networks (DNNs) in SRCNN <cit.>, subsequent studies <cit.> have significantly advanced SR performance beyond traditional methods. In the specific context of omnidirectional image super-resolution (ODISR), DNN-based approaches have been tailored to account for the unique latitude-based characteristics of ODIs <cit.>. For example, LAU-Net <cit.> divides the entire ERP image into latitude-based patches for separate upscaling. SphereSR <cit.> introduces the spherical local implicit image function (SLIIF) alongside a novel feature extraction module to leverage information from arbitrary projection types. OSRT <cit.> employs a distortion-aware transformer targeting dimension-related distortions in ERP images. OPDN <cit.> introduces a dual-stage framework incorporating a position-aware deformable network. Despite these noteworthy advancements, the majority of these methodologies presuppose a fixed downscaling approach (e.g. bicubic <cit.>) and overlook high-frequency components from HR inputs, thus limiting the quality of reconstructed details in SR ODIs.




 §.§ Image Rescaling

Different from SR, image rescaling focuses on downscaling HR images to create visually pleasing LR images that retain essential information for accurate HR reconstruction. Recently, invertible neural network (INN) <cit.> becomes a representative framework for image rescaling <cit.>, offering a direct route to inversely map the downscaled images back to HR. For instance, IRN <cit.> is the first attempt to model image downscaling and upscaling using invertible transmissions. Liang et al. <cit.> formulate high-frequency components in INNs as a conditional distribution on the LR image. HyberThumbnail <cit.> employs an asymmetric encoder-decoder architecture for real-time reconstruction of 6K images and also optimizes the JPEG compression process <cit.>. Very recently, DINN <cit.> makes the first attempt to apply image rescaling to ODI and highlights the significance of leveraging ERP's latitude characteristics by developing a latitude-aware conditional mechanism. However, existing ODI rescaling methods focus solely on improving the quality of ERP images. In contrast, our ResVR innovatively optimizes the quality of the final viewport, offering new improvements orthogonal to previous ODI rescaling methodologies.






 §.§ Viewport Rendering of ODIs

ODIs are designed to encapsulate a full spherical view, enabling an immersive viewing experience. However, when viewed through HMDs towards a particular direction, only a specific viewport is displayed <cit.>. Achieving high resolution and quality in these viewports is crucial for immersive experiences, as highlighted by various ODI visual quality assessment techniques <cit.>. As a widely adopted method for viewport rendering <cit.>, perspective projection employs a series of pixel mapping and resampling operations to effectively implement image warping. To reduce the interpolation-induced blurriness and artifacts, SRWarp <cit.> reinterprets image warping as an SR problem and introduces a differentiable warping module. LTEW <cit.> uses a continuous neural representation <cit.> for image warping by taking advantage of both Fourier features and spatially-varying Jacobian matrices. LeRF <cit.> assigns spatially varying steerable resampling functions to pixels, learning their orientations for continuous function prediction. Different from the above methods that focus on warping, our ResVR considers comprehensive ODI processing, serving as a new framework while enjoying high viewpoint rendering quality.








§ METHODOLOGY

In this section, we begin with a concise review of the viewport rendering process (Sec. <ref>). Following this, we provide an overview of our ResVR framework (Sec. <ref>), which is illustrated in Fig. <ref>. We then elaborate on the proposed discrete pixel sampling strategy (Sec. <ref>) and spherical pixel shape representation technique (Sec. <ref>). The training objectives are detailed in Sec. <ref>.





 §.§ Preliminaries of Viewport Rendering


ODIs inherently provide a full spherical view. However, when viewed through HMDs directed towards a specific direction, only the corresponding viewport is displayed. This viewport appears as a 2D image, derived through perspective projection <cit.> from a segment of the spherical image. To formalize this process, we consider the view direction in spherical coordinates (θ_c, ϕ_c), along with the horizontal and vertical fields of view (F_h, F_v), and the height and width of the viewport (h_v, w_v). An invertible coordinate mapping f: ↦ is established, where :={|∈^2} denotes the coordinate space of ERP, and :={|∈^2} represents the coordinate space of the viewport to be rendered. In practice, the coordinates _view on the viewport are initially determined by (h_v, w_v), and then the corresponding coordinates on ERP are obtained through backward mapping _view = f^-1(_view). The rendering of the viewport is achieved through resampling techniques, such as interpolation, ensuring that this process remains fully differentiable. Additional mathematical details of perspective projection and viewport rendering are elaborated in the supplementary material.





 §.§ Overview of ResVR



An overview of proposed joint Rescaling and Viewport Rendering (ResVR) of ODIs is presented in Fig. <ref>. The ODI processing of ResVR contains two steps: (1) ODI Downscaling & Compression: An HR ERP image is firstly downsampled and compressed to an LR ERP JPEG image for efficient transmission from cloud server to user HMDs, and (2) Decompression & Viewport Rendering: The LR ERP image is then decompressed and rendered to HR viewports on HMDs through our viewport rendering (VR) module.


ODI Downscaling & Compression: Given an HR ERP image ∈^3× H × W, its LR representation is firstly generated through our downsampler, where s is the rescaling factor. The downsampler is a U-Net <cit.> with dense blocks <cit.>. To further decrease the size of the transmitted image file, a JPEG encoder is employed for the LR representation to obtain a JPEG image. Concretely, we follow <cit.> to predict adaptive quantization tables for each image. Finally, the LR JPEG image is obtained, and adaptive quantization tables and quantized DCT coefficients are also encoded into the JPEG file. More details about the downsampler, the adaptive quantization table prediction module, and the training process of learned compression are provided in the supplementary material.









Decompression & Viewport Rendering: After receiving the LR JPEG image, ∈^3 ×H/s×W/s is firstly reconstructed by the JPEG decoder through inverse discrete cosine transformation (IDCT). Then our goal is to render high-resolution viewport  ∈^3× h_v× w_v directly from . Inspired by recent implicit neural representation methods <cit.>, we develop a viewport rendering (VR) module to predict pixel values of the query coordinates _view from the latent space of input  instead of using traditional interpolation methods. As depicted in Fig. <ref>, our VR module consists of an encoder ℰ, a local texture estimator h_ψ={h_a, h_f, h_p}, and an MLP decoder 𝒟. h_ψ is a learnable dominant-frequency estimator, which is capable of characterizing image textures in 2D Fourier space <cit.>. Here, h_a is an amplitude estimator (^C ↦^256), h_f is a frequency estimator (^C ↦^2× 128), and h_p is a phase estimator (^10↦^128). Concretely, for a query point ∈_view, the estimating function h_ψ is defined as:



    h_ψ(_j,δ_,())=h_a(_j) ⊗[
            cos{π(<h_f(_j),δ_>+h_p(())) }
    sin{π(<h_f(_j),δ_>+h_p(())) }],

where =ℰ(). Denote j as a pixel index of the , _j and _j are the corresponding coordinates and latent variable of , respectively. δ_ is a local grid calculated by δ_ = -_j. <·,·> is an inner product, and ⊗ denotes element-wise multiplication. To be noted, () is the spherical pixel shape representation of the query coordinate , which is important for providing spatial-varying priors <cit.> for our VR module and will be elaborated on in Sec. <ref>.
Finally, our VR module predicts the RGB values of a coordinate =f() on the viewport as:

    [] = ℱ_B() + ∑_j∈𝒥ω_j𝒟(h_ψ(_j, δ_, ())),

where ℱ_B is a bilinear interpolation operator to stabilize the convergence of network training and aid the VR module in learning high-frequency details. 𝒥 is a neighborhood set of , defined as 𝒥={j|j=+[ms/W, ns/H], m,n∈{-1,1}}, while ω_j is a local ensemble coefficient. More details of our VR module are provided in the supplementary material. Different from existing ODI rescaling methods which first produce  and then obtain HR viewports through traditional interpolation methods, our ResVR directly renders the final HR viewports from  through our VR module.



 





 §.§ Discrete Pixel Sampling Strategy




Motivation. In image rescaling and SR tasks, the ground truth HR images  are typically cropped into patches  to reduce GPU memory overhead while increasing the diversity of training data. The goals of our ResVR's end-to-end training process include: (1) optimizing the downscaling process ( ↦ ), and (2) optimizing the rendering process ( ↦ ). In this framework, the former process requires the corresponding  as supervision, while the latter process needs the corresponding  as supervision. However, due to the natural geometric properties of ODIs' different projection types, there exists a challenge of shape mismatch between a viewport and its corresponding ERP area. As depicted in Situations #1 and #2 in Fig. <ref>, one can not obtain  and  that are both with rectangular image shape at the same time. As a result, conventional image-level loss can not be directly used for end-to-end training of ResVR. To address this, we innovatively propose a discrete pixel sampling strategy (DPS), as shown in “Ours” of Fig. <ref>.



Method. Denoting p as both the height and width of training patches, our core idea is to keep the LR ERP patch with a rectangular image shape (∈^3×p/s×p/s ), while discretely sampling N pixels S_pix∈^3× N with their coordinates _samp∈^2× N in the irregular area. Thanks to the continuous representation ability of implicit neural representation methods <cit.>, our VR module is able to predict corresponding pixel values Ŝ_pix given coordinates _samp. Hence, we use the sampled S_pix as supervision for the predicted Ŝ_pix. An overview of our strategy is presented in Algo. <ref>.


Specifically, given the left top coordinate (a, b) of the training patch, the view direction (θ_c, ϕ_c), FoVs (F_h and F_v), shapes (h_v, w_v), and query coordinates (_view) of the desired viewport, we first determine the coordinate mapping f. Then the corresponding coordinates _view in the ERP space can be obtained through inverse coordinate mapping as _view = f^-1(_view). By setting appropriate view direction and FoVs, it is possible to ensure that some elements in _view lie in the area of the cropped ERP patch. Recall that p is the size of the cropped training patch, _view is then filtered to preserve the subset overlapped with the patch as follows: 

    _view'={∈_view|a≤_1<(a+p), b≤_2<(b+p)},

where _1 and _2 represent the coordinates of a point  in the ERP space. This filter operation ensures that every point in _view' can find its corresponding latent variable extracted from . To balance the number of pixels sampled from different FoVs and view directions, if the filtered _view' contains more than N pixel coordinates, only N elements will be randomly retained. Finally, to uniform the coordinate correspondence between LR and HR patches, _view'⊆ [0, H)× [0, W) is converted from the whole HR ERP coordinate space to the patch coordinate space, and normalized to _samp⊆ [-1,1)×[-1,1), as:

    _samp   = {T()|∈_view'}, 
    where  T()   =2(((_1, _2)-(a,b))/p) - 1.

As a result, S_pix are sampled through bicubic interpolation as S_pix = ℱ_bicubic (, _samp). By adopting Algo. <ref>, ∈^3×p/s×p/s can be used as supervision of the predicted  , and S_pix∈^3× N can be used as supervision of the predicted Ŝ_pix.







 §.§ Spherical Pixel Shape Representation





Motivation. Pixel shape representations described by grid orientation and curvature provide informative geometric spatial-varying priors for image warping tasks <cit.>. Existing shape representation methods are designed for 2D images. However, different from 2D images, ERP is an unfolding of the spherical surface along meridians, which leads to the fact that the adjacent pixels on the sphere can be far apart in the ERP image. As a result, the previous 2D shape representation methods fall in the high-latitude/-longitude areas due to the nature properties of (1) latitude-related distortion and (2) wraparound consistency of ERP images (Fig. <ref> in Sec. <ref>). Therefore, we develop a spherical pixel shape representation (SSR) technique to solve this challenge. As Fig. <ref> shows, SSR leverages the information of transformed coordinates on the sphere as a more effective shape representation to guide the viewport rendering process.

Method. In image warping tasks, the first-order partial derivatives (i.e. Jacobian matrix) and the second-order partial derivatives (i.e. Hessian matrix), describe the orientation and curvature of pixels resulting from the transformation, respectively <cit.>. These two matrices provide informative geometric spatial-varying priors to guide the process of viewport rendering. Inspired by this, for a point  on the original ERP plane, we represent its pixel shape () with the gradient of the inverse coordinate transformation f^-1. Specifically, we start with the point =f() on the viewport plane, and the corresponding Jacobian matrix 𝐉_f^-1() and Hessian matrix 𝐇_f^-1() are analytically computed as:

    𝐉_f^-1() =
         [
            ∂_1/∂ u ∂_1/∂ v
    ∂_2/∂ u ∂_2/∂ v],  𝐇_f^-1() =
         [
            ∂^2 _1/∂^2 u ∂^2 _1/∂ u ∂ v
    ∂^2 _2/∂ v ∂ u ∂^2 _2/∂^2 v],

where =f^-1()=(_1, _2) represents the coordinates in the original ERP plane. We propose a spherical pixel shape representation (SSR) technique to numerically estimate 𝐉_f^-1() and 𝐇_f^-1() based on spherical differentiation. Specifically, as depicted in Fig. <ref>, the inverse coordinate transformation f^-1 is firstly applied to  and its eight nearest points ( + [m/w_v, n/h_v] with m,n∈{-1,0,1}) to get  and its neighborhood on the ERP plane. Different from existing 2D shape representation methods <cit.> which estimate the shape of  directly on the ERP image plane, we further transform  and its neighborhood to the sphere denoted by the set {_i|_i=(θ_i,ϕ_i), i=1,2,…,9}, and innovatively calculate numerical derivatives to estimate 𝐉_f^-1() and 𝐇_f^-1() by the proposed spherical central difference method, overriding Eq. (<ref>) as:

    𝐉_f^-1()    ≈[
            
                D(_6, _4) 
    
                D(_2, _8)
            ], 
    𝐇_f^-1()    ≈[
            
                D(_6, _5) + D(_5, _4)    D(_3, _1) + D(_9, _7)
    
                D(_3, _1) + D(_9, _7)   D(_2, _5) + D(_5, _8)
            ],

where D(_i,_j) calculates the distance between _i and _j in the spherical coordinate system as:

    D(_i, _j) = (
    min(|ϕ_j-ϕ_i|,     2π-|ϕ_j-ϕ_i|)
    min(|θ_j-θ_i|,  2π-|   θ_j-θ_i|)·cos((θ_i+θ_j)/2)
    )^⊤,

where |·| represents the absolute value operation. The min(·) operator ensures the computation of minor arc distances, and the average latitude is utilized to adjust the longitudinal distance to account for the convergence of meridians. We follow <cit.> to use six elements in 𝐇_f^-1(), and finally ()∈^10 is obtained by concatenating and flattening 𝐉_f^-1() and 𝐇_f^-1(). As a result, () serves as an auxiliary input, together with the LR latent representation =ℰ() to predict final viewports through the VR module.







 §.§ Training Objectives


Thanks to the discrete pixel sampling strategy (Sec. <ref>) and the fully differentiable pipeline (Fig. <ref>), the processes of downscaling, compression, and viewport rendering can be jointly optimized end-to-end, which aligns the learning of network parameters with our goals of obtaining high-quality viewport while reducing the transmission overhead. The total loss is a weighted sum of a pixel-level reconstruction loss, an LR guidance loss, and a bitrate loss as:

    ℒ=ℒ_pix + λ_1ℒ_guide + λ_2ℒ_bpp,

where λ_1 and λ_2 are two trade-off parameters.

Pixel-level reconstruction and guidance loss. In the training phase, we employ Algo. <ref> to get S_pix as the ground truth and use the VR module to predict the corresponding values Ŝ_pix given X_samp. 

    ℒ_pix = ||Ŝ_pix- S_pix||_1/N,

where N is the number of sampled pixels. Additionally, following <cit.>, an L_2 guidance loss on the LR ERP patch is defined as:

    ℒ_guide = || - ||^2_2/(p/s) × (p/s).


Bitrate loss. To optimize the size of the transmitted JPEG image, we firstly follow <cit.> to estimate the rate R of the quantized coefficients C with differentiable fully factorized entropy models as: R=𝔼_x∼ p_x[-log p_L(C_y)-log p_C(C_Cb)-log p_C(C_Cr)], where p_L and p_C are two fully factorized entropy models for luma and chroma coefficient maps, respectively. Then the bpp of the transmitted LR ERP JPEG image is calculated as:


    ℒ_bpp=R/H× W.









§ EXPERIMENTS



 §.§ Experimental Setup

Implementation details. We follow the previous work <cit.> to put more computation into the downsampler to keep the VR module lightweight. The VR module is composed of a lightweight feature extractor and a tiny MLP. Please refer to our supplementary materials for more details of our network architecture.

Training details. The downscaling factor s is fixed to 4, and patch size p of random cropped  is set to 256. To ensure the overlapping of  and discretely sampled points in Algo. <ref>, we calculate the viewport center (θ_c, ϕ_c) according to the center of cropped patch. To enable ResVR to handle different resolutions and FoVs, we randomly sample FoVs and resolutions during training. Specifically, the FoVs (F_h, F_v) are randomly sampled from {80^∘, 90^∘, 100^∘, 110^∘, 120^∘} and the resolutions of viewport (h_v, w_v) are randomly sampled from {512, 576, 640, 768, 832, 960, 1024}. The number of sampled pixels N is set to 25600, and the batch size is set to 16. All experiments are conducted on one V100 GPU. The network is trained for 5× 10^5 iterations with learning rate 2× 10^-4. λ_1 and λ_2 are set to 0.6 and 0.01 for all experiments, respectively.


Datasets and evaluation metrics. ODI-SR dataset <cit.> and SUN360 Panorama dataset <cit.> are used in our experiment. We follow the data split setting in <cit.> and train on the ODI-SR training set. For evaluation of transmission efficiency, we follow <cit.> to use the real file size of JPEG for evaluating the bitrate: bpp=𝔼_x∼ p_x[filesize/(H× W)]. For evaluation of the quality of rendered viewports, we choose ten different view directions, get the ground truth image using bicubic interpolation, and calculate the average PSNR, SSIM, and LPIPS <cit.>. More details are provided in the supplementary materials. For those competing methods that need to explicitly produce HR ERP images to render final viewports, we evaluate the WS-PSNR <cit.> of their predicted .





 §.§ Comparison with State-of-the-Art Methods


We compare three categories of methods: (1) a baseline method which downscales with Bicubic interpolation, compresses with standard JPEG codec, and renders viewports with Bicubic interpolation; (2) SR pipeline which downscales with Bicubic, compresses with standard JPEG codec, upscales with state-of-the-art methods <cit.>, and uses bicubic interpolation for viewport rendering; (3) Rescaling pipeline which firstly uses state-of-the-art asymmetric rescaling methods <cit.> and renders viewports with Bicubic interpolation. For a fair comparison, we retrain SR methods <cit.> and rescaling methods <cit.> with our training sets and constrain their bpp around 0.3 by adjusting the quality factor of JPEG compression in all baselines.

Tab. <ref> presents the quantitative comparisons of the quality of rendered viewports with (F_h, F_v)=(120^∘,90^∘) and (w_v, h_v)=(2048, 1536) among different methods. Taking advantage of directly optimizing the final viewport through end-to-end training, ResVR outperforms previous methods by reconstruction accuracy (about 0.4dB gain on PSNR) and with better realness (about 0.02 gain on LPIPS). Notably, even though existing rescaling methods can achieve PSNR of about 28dB, they only obtain sub-optimal results due to the lack of awareness and optimization of the viewport rendering process, which demonstrates the fact that focusing solely on the quality of ERP images results in sub-optimal visual experience of viewports. ResVR directly optimizes the rendered viewports without the need to predict HR ERP and achieves SOTA performance. Fig. <ref> further provides the visual comparison of rendered viewports. It can be seen that our ResVR reconstructs enhanced details with fewer artifacts (e.g. the text “labatt's” in the top image and the lines in the bottom image) compared to other methods. Additionally, ResVR shows better wraparound continuity at the location of the seams (e.g., pipe in the zoomed-in area) in the bottom case. We attribute this to the awareness of arbitrary view directions during training and our MLP’s continuous representation of images.



 §.§ Ablation Study


In this section, we analyze the effect of the proposed discrete pixel sampling strategy and spherical pixel shape representation. We conduct experiments on three ResVR variants: (1) Case # 1: We train the rescaling process and VR module separately with 2D shape representation; (2) Case # 2: We train the whole pipeline in an end-to-end manner by employing discrete pixel sampling strategy with 2D shape representation; (3) The complete ResVR with our sphere pixel shape representation. Quantitative results are shown in Tab. <ref>.

Effect of discrete pixel sampling strategy. Comparison # 1 vs. # 2 exhibits the effectiveness of the proposed discrete pixel sampling strategy which enables end-to-end training of the whole pipeline, thus providing substantial PSNR improvement of 4.12-4.44dB. To further analyze the sampled coordinates and pixels, we visualize the sampled areas and pixels in the supplementary materials.





Effect of spherical pixel shape representation. Comparison # 2 vs. ResVR reveals that the spherical shape representation improves PSNR by 0.40-0.55dB. We observe that conventional 2D shape representation methods <cit.> fall in high-longitude and high-latitude areas, as depicted in Fig. <ref>. We attribute this to the following reasons: adjacent pixels on the viewport should be very close to each other on the sphere. However, (1) for high-latitude areas, due to the natural distortion of ERP, the distance between corresponding points is very close to the original sphere but is very far on the ERP image. (2) For high-longitude areas, due to the wraparound consistency of the left and right ends of ERP, the corresponding points are originally adjacent pixels on the sphere but are at both ends of the ERP image. The traditional shape representation method based on 2D ERP ignores the spherical characteristics, so in the above situations, it provides wrong pixel shape priors for the VR module, thus leading to abnormal visual results. The proposed SSR is performed on the native spherical surface, thus ensuring stable and high-quality rendering quality in various view directions.







 §.§ Analysis

LR JPEG image. The file size and visual quality of LR JPEG images are also important for efficient transmission and user preview. ResVR effectively balances transmission efficiency with the final viewer experience through our end-to-end training. Visualizations of LR images are provided in the supplementary materials.


Arbitrary-resolution viewport rendering. Due to the continuous representation ability <cit.> of the MLP in our VR module, ResVR is able to render viewports of different resolutions using only one set of parameters. Tab. <ref> compares ResVR with SOTA methods on different resolution settings. It can be seen that ResVR achieves better performance at different resolutions.




§ CONCLUSION

The rise of virtual reality and augmented reality applications has popularized ODI rescaling to shrink the file size of images while maintaining their quality. However, existing methods focus on improving the ERP image quality, ignoring that HMDs use rendered viewports for display, not ERP images. This focus on ERP quality alone will lead to compromised user experiences. To address this, in this paper, we propose ResVR, a novel framework for joint rescaling and viewport rendering of ODIs. In ResVR, we develop a discrete pixel sampling strategy to tackle the irregular correspondence between the ERP area and the viewport, enabling end-to-ending training of the whole pipeline. A spherical pixel shape representation technique is introduced to serve as an effective guidance for the rendering process, further enhancing the visual quality of viewports. Experiments demonstrate that our ResVR achieves state-of-the-art performance across different settings of FoVs, view directions, and resolutions while keeping a low transmission overhead.



Limitations and future work. Existing ODI datasets are of relatively low resolution and with compression artifacts, which limits the further performance improvements of ResVR. Besides, although ResVR provides a promising pipeline for comprehensive ODI processing and achieves SOTA performance, it is crucial to extend ResVR to 360^∘ video tasks <cit.>. We leave this for future work.


ACM-Reference-Format








