ESR-NeRF: Emissive Source Reconstruction Using LDR Multi-view Images
Jinseo Jeong Junseo Koo Qimeng Zhang Gunhee Kim

First detection of the [CII] 158 μm line in the intermediate-velocity cloud Draco
    Nicola Schneider     1
Volker Ossenkopf-Okada  1 
Eduard Keilmann         1
Markus Röllig         2,1 
Slawa Kabanovic         1 
Lars Bonne              3    
Timea Csengeri          4    
Bernd Klein             5,6    
Robert Simon            1
Fernando Comerón      7

    draft of April 29, 2024
====================================================================================================================================================================================================================================================================================


    
    Existing NeRF-based inverse rendering methods suppose that scenes are exclusively illuminated by distant light sources, neglecting the potential influence of emissive sources within a scene.
    
    In this work, we confront this limitation using LDR multi-view images captured with emissive sources turned on and off.
    
    Two key issues must be addressed: 1) ambiguity arising from the limited dynamic range along with unknown lighting details, and 2) the expensive computational cost in volume rendering to backtrace the paths leading to final object colors.
    
    We present a novel approach, , leveraging neural networks as learnable functions to represent ray-traced fields.
    
    By training networks to satisfy light transport segments, we regulate outgoing radiances, progressively identifying emissive sources while being aware of reflection areas.
    
    The results on scenes encompassing emissive sources with various properties demonstrate the superiority of in qualitative and quantitative ways.
    
    Our approach also extends its applicability to the scenes devoid of emissive sources, achieving lower CD metrics on the DTU dataset.





§ INTRODUCTION


Extensive research has focused on reconstructing 3D object structures <cit.>, material properties <cit.>, and lighting <cit.> from 2D images, applicable across domains including 3D graphics and augmented reality <cit.>.

This endeavor not only facilitates the creation of life-like virtual objects but also streamlines the process of scene manipulation <cit.>.
Recent advancements <cit.> have built on Neural Radiance Fields (NeRF) <cit.> successes in novel view synthesis <cit.>.

Significant progress in re-lighting <cit.> has facilitated scene editing via manipulating the reconstructed light sources.

However, existing methods predominantly deal with the scenes lit by distant sources, like environment maps or collocated flashlights.

Notably, NeRF-based inverse rendering has yet to consider scenes with multiple emissive sources, a common real-world illumination condition.


Emissive sources in a scene introduce critical challenges: (i) ambiguity in decomposing scene components and (ii) high computational costs for analyzing the causes of pixel colors.

This ambiguity stems from difficulties in identifying emissive source regions, as illustrated in Fig. <ref>.

Contrary to prior setups <cit.>, we allow the possibility of numerous emissive sources throughout the scene.

In standard photographs with pixel values from 0 to 255, the distinction between emissive sources and nearby reflection areas is challenging.


As shown in Fig. <ref>, relying solely on pixel value thresholding is insufficient for differentiating between emissive sources and their reflections.

Naive inverse path tracing is impractical, due to the computational costs rising exponentially with the number of ray bounces in volume rendering.

This can cause inaccuracy in emissive source reconstruction, yielding unrealistic illumination in reflective areas as users manipulate emissive sources.




To address these challenges, we introduce (Emissive Sources Reconstructing NeRF), a novel approach capable of reconstructing any number of emissive sources by progressively discovering reflection areas. 


We assume that the scenes are observed in two lighting conditions: one with all emissive sources active and the other with them inactive.
Our approach utilizes neural networks as learnable functions for representing ray-traced fields.

By training networks to satisfy each light transport segment, we sidestep the computational overhead of ray tracing associated with ray bounces.


In this work, we exclusively use low dynamic range (LDR) images, setting us apart from prior mesh-based methods that rely on high dynamic range (HDR) images <cit.>.

Our experiments encompass synthetic and real scenes, ranging from single to multiple lighting configurations with complex reflections.
The scenes vary in light source counts, color, and intensity.

Qualitative and quantitative evaluations show 's superiority over state-of-the-art NeRF-based re-lighting methods.

Furthermore, Chamfer Distance (CD) metrics on the DTU dataset <cit.> indicate 's competitive performance in scene reconstruction, even without emissive sources. 


We summarize our contributions as follows.


    
    
  * Our work presents the first NeRF-based inverse rendering that can deal with the scenes with any number of emissive sources, challenging the distant light assumption of previous research.
    
    
  * Unlike existing mesh-based methods relying on HDR images, we use LDR images for the first time, overcoming the poor representation of emissive sources.
    
    
  * We provide a benchmark dataset designed to evaluate the performance of emissive source reconstruction.
    
    
  * Our method is applicable to the scenes with or without emissive sources, achieving superior mesh reconstruction results on the DTU dataset.





§ RELATED WORK

































Neural Rendering.

Advancements in implicit representations <cit.> and volume rendering <cit.> have significantly enhanced neural rendering capabilities, enabling the reconstruction of scene components from 2D images.

One of the key directions is mesh extraction <cit.>, with methods like NeuS <cit.> and VolSDF <cit.> utilizing signed distance function (SDF) values for volume rendering.




Recently, the efficient computation of volume rendering has become a focal point due to the substantial computational cost associated with network inference for ray color calculation <cit.>.

Several methods propose to directly predict ray color using the 4D light fields concept <cit.> or leveraging voxel grids for fast inference of spatial features <cit.>.

NeuralRadiosity <cit.> shares similarity with our method, as it predicts ray-traced values instead of explicitly tracing individual rays.
However, they primarily focus on calculating the final object color when all scene information is available.

In contrast, our inverse rendering approach aims to reconstruct emissive sources within a scene, addressing the ambiguities introduced by their presence in LDR images.




Inverse Rendering.
A growing emphasis revolves around the decomposition of materials represented by spatially varying bidirectional reflectance distribution functions (SVBRDF) <cit.>.




To lessen the computational burden in inverse rendering <cit.>, several methods have adopted neural networks as lookup tables <cit.> or computational caches <cit.>.







While NeRV <cit.> utilizes caching visibility and NeILF++ <cit.> adopts caching surface point radiance with the inter-reflection loss for incident radiance, 

our method diverges by focusing on tracing radiance origins. Specifically, we aim to identify emissive sources within a scene, moving beyond the simplification of incident radiance calculations.






Several methods rely on diverse known lighting configurations to exploit variations in object appearances <cit.>.
Toggling emissive sources on and off resembles the common one-light-at-a-time (OLAT) technique, as seen in NLT <cit.> and ReNeRF <cit.>. However, our setting does not need to know light source properties and to toggle lights individually. 
Instead, we allow for toggling all lights together.
Recent works have also jointly reconstruct the mesh, materials, and lighting <cit.>.
They tackle with images captured under a single unknown lighting condition <cit.>, assuming that radiance already encodes global illumination <cit.>.
However, they confine to the scenes illuminated by far-distant lights, constrained to an 8-bit color spectrum.



Our work considers the presence of multiple emissive sources within a scene captured in LDR images, questioning the prevailing notion that radiance fields trained with the image rendering loss faithfully represents global illumination.


While some methods <cit.> deal with the scenes featuring emissive sources, they work outside the volume rendering framework and depend on HDR input images, assuming prior knowledge of scene geometry.







§ PRELIMINARIES






Surface Representation.
Analgous to NeRF <cit.>, neural network f_θ predicts SDF values at arbitrary 3D spatial locations.

NeuS <cit.> integrates surface representation into volume rendering using the SDF-based opacity ρ(x)=max(-dΦ_s/dx(f(x))/Φ_s(f(x)),0).

Here Φ_s(x)=(1+e^-sx)^-1 is the sigmoid function where s controls the sharpness of surfaces.

The color of a ray can be calculated as





    Ĉ(r) = ∫_0^∞ T(r(t))ρ(r(t)) L_o(r(t),ω_o)  dt,




where Ĉ(r) denotes the predicted ray color, r(t;c,ω_o)=c-t·ω_o is the ray with camera center c along direction ω_o, T(r(t)) = exp(∫_0^t -ρ(r(u)) du) is the transmittance, and L_o(r(t),ω_o) is the outgoing radiance. 
Henceforth, we use x to denote a point in r(t;c,ω_o) for notational simplicity. 



Light Transport in Volume Rendering.
Extracting light sources necessitates analyzing the causes affecting the final ray colors.


Kajiya's rendering equation <cit.> factorizes the outgoing radiance L_o(x,ω_o) into emission and reflections:




    L_o(x,ω_o)=E(x)+∫_ΩL_i(x,ω_i) R(x,ω_o,ω_i;b) dω_i,

where E(x) is the emission, R(x,ω_o,ω_i;b) represents the SVBRDF parametrized by parameters b with Lambert cosine multiplied, and L_i(x,ω_i) is the incident radiance.
In volume rendering, computing the incident radiance at point x is akin to evaluating Eq. <ref>, with x serving as the camera center.


By iteratively factorizing the outgoing radiance in the incident radiance, the contribution of a path length i for a pixel can be decomposed as in Eq. <ref>,





where ℋ_i=Π_j=1^i-1T(x_j)ρ(x_j)R(x_j,ω_j-1,ω_j) is the path throughput, S(ω_i) is the environment map strength in direction ω_i, and V(x,ω_i)=exp(∫_0^∞ -ρ(r(u;x_i,-ω_i)) du) is the visibility of the environment map at point x along direction ω_i:



    P_i    = ∫_l_1∫_Ω⋯∫_l_i-1∫_Ω (∫_l_iT(x_i)ρ(x_i) E(x_i)  dt_i + 
        S(ω_i-1)V(x_i-1,ω_i-1)) ℋ_idt_1dω_1⋯ dt_i-1dω_i-1.


Extending the analysis to longer light paths, or equivalently, increasing the number of ray bounces, leads to exponential growth in computation complexity.

This poses a challenge when attempting to decompose the influence of unknown emissive sources, as their ability to produce strong reflections makes ignoring indirect illumination infeasible.

 


§ METHODOLOGY






None of the previous works address the reconstruction of emissive sources from LDR multi-view images.
Sec.  <ref> through  <ref> detail our method, , which reconstructs emissive sources without prior knowledge of scene geometry, materials, or lighting specifics (including their location, number, or colors).

We also show how these reconstructed sources can be used for scene editing in  <ref>.








 §.§ Learnable Tone-mapper



Throughout the paper, we use ℛ to represent camera rays,
C for pixel values, and a binary flag 𝕀 to indicate
whether an image is captured with emissive sources on or off.













To extract HDR values from LDR images, we employ the softplus activation for outgoing radiance prediction and apply a clipping and gamma function τ <cit.> for the rendering loss such that Ĉ_τ(r) = τ(Ĉ(r)).
Unlike previous NeRF-based works <cit.> that limit radiance to the range of [0,1], our approach allows for any positive radiance values.

Yet, it creates difficulties in differentiating between the surface weight T(x)ρ(x) and the magnitude of radiance value L_o(x,ω_o),
since it allows for the possibility of assigning extreme radiance to
the points with low surface weights to render same ray colors.








Such ambiguity poses challenges, particularly in dark and high-contrast scenes, aggravating surface reconstruction (see Fig. <ref>).

To address this, we introduce a learnable tone-mapper m_θ: ℝ^3_+→[0,1]^3, that takes positionally encoded HDR linear values as input:


    Ĉ_m_θ(r) = ∫_0^∞ T(x)ρ(x) m_θ(L_o(x,ω_o))  dt,


    L_o(x,ω_o) = L_o^S(x,ω_o) + L_o^E(x,ω_o) ·𝕀,


where L_o^S(x,ω_o) is radiance when emissive sources are turned off, while L_o^E(x,ω_o) stands for radiance added to the scene by emissive sources.









Our rendering loss is then formulated as follows, with λ_τ as a hyper-parameter:

    ℒ_render=∑_r ∈ℛ (‖ C(r) - Ĉ_m_θ(r)‖_2^2 + λ_τ‖ C(r) - Ĉ_τ(r)‖_2^2 ).
















 §.§ Learning of Light Transport Segments



The computational complexity of object appearance analysis in volume rendering is notably high, as shown in Eq. <ref>.

We take an alternative approach by leveraging neural networks to represent ray-traced fields, rather than explicitly tracing every rays.
Our distinct contribution to inverse rendering lies in precise adjustment of radiance.














Specifically, we impose constraints on the predicted radiance to satisfy each light transport segments.
The light transport segments (LTS) loss, ℒ_lts, plays a pivotal role in our method:

    ℒ_lts^S = ∑_x,ω_o‖ L_o^S(x,ω_o) - L̂_o^S(x,ω_o) ‖_2^2,


    ℒ_lts^E = ∑_x,ω_o‖ L_o^E(x,ω_o) - L̂_o^E(x,ω_o) ‖_2^2,


    L̂_o^S(x,ω_o) =  ∫_Ω S(ω_i) V(x, ω_i) R(x, ω_o, ω_i)_direct illumination by an environment map dω_i  +     
       ∫_Ω∫_0^∞T(x')ρ(x')L_o^S(x',-ω_i)  dt 'R(x, ω_o, ω_i)_indirect illumination by an environment map dω_i.





    L̂_o^E(x,ω_o) = E(x)_emission +                                                                                                                       
       ∫_Ω∫_0^∞T(x')ρ(x')L_o^E(x',-ω_i)  dt' R(x, ω_o, ω_i)_direct & indirect illumination by emissive sources dω_i.





We ensure consistency between the radiance directly predicted by the network L_o(x,ω_o) and the radiance achievable based on the scene context L̂_o(x,ω_o).
Previous approaches have focused on matching L̂_o(x,ω_o)
to training views, overlooking the relations to L_o(x,ω_o).
This hinders the restoration of HDR radiance by supervising scene components to LDR training views.


In contrast, our LTS loss enables volumetric energy transfer of radiance, adjusting outgoing radiance based on their interrelations.





To implement this concept, we train six dedicated networks for SDF f(x), SVBRDF parameters b(x), emission E(x), environment map S(ω_i), outgoing radiances L^S_o(x,ω_o) and L^E_o(x,ω_o), to adhere to these LTS requirements.
For the environment map, we represent it using 48 Spherical Gaussians <cit.> : ∑_k=1^Mμ_k e^λ_k(ω_i·ξ_k -1), followed by the softplus activation.
μ∈ℝ^3, λ∈ℝ_+, and ξ∈𝕊^2 respectively denote the lobe amplitude, sharpness, and axis.




 §.§ Progressive Discovery of Reflection Areas



l0.48
    

    0.48
        
    < g r a p h i c s >

    
    0.48
        
    < g r a p h i c s >

    
    Left: Image with active emissive sources. Right: Identified emissive sources w/o progressive discovery of reflection areas.
    
    



Relying solely on LTS is insufficient for addressing ambiguity arising
from low pixel values of emissive sources and intense reflections in
adjacent regions, often leading to confusion between emission and reflection.








The right image in Fig. <ref> shows self-emitting objects restored with the naive LTS loss.
While emissive sources are small, large areas affected by them are also identified as emissive sources.
We propose a reflection-aware progressive approach for precise identification of emissive sources.
By leveraging LTS learning, we extend the regions that can be regarded as reflection areas.
Fig. <ref> illustrates our progressive algorithm.



Reflection-Aware Emission Refinement.


Since surface points are unknown and are updated during learning, we opt to utilize rays rather than surface points.
This process involves categorizing training rays into two groups: uncertain (ℛ^U) and certain (ℛ^C).
The certain group contains the rays confidently identified as reflection, aiding the transfer of radiance energy to nearby points.


For the points in the certain group, we use the Eq. <ref> instead of Eq. <ref> to exclusively attibute outgoing radiances to reflections.
Satisfying the LTS loss on the certain group results in adjusting the outgoing radiances of influential points, as illustrated in Fig. <ref>(a):

    L̂_o^E(x,ω_o) = ∫_Ω∫_0^∞T(x')ρ(x')L_o^E(x',-ω_i)  dt' R(x, ω_o, ω_i)  dω_i.







The uncertain group includes the rays indicating the areas that are undetermined yet as reflection or emission.

Using Eq. <ref> to compute L̂_o^E(x,ω_o), this group adjusts emissions E(x) based on the radiance updates by the certain group, where “sg” represents the stop-gradient:


    L̂_o^E(x,ω_o) = E(x) +                                                                                                  
       sg(∫_Ω∫_0^∞T(x')ρ(x')L_o^E(x',-ω_i)  dt' R(x, ω_o, ω_i)  dω_i).








As shown in Fig. <ref>(b), this leads to increased emissions for the regions whose radiances are adjusted to account for the reflections in the certain group.

Conversely, emissions decrease for the regions where there is little change in outgoing radiance, but incident radiances are increased by surrounding influential points.







Ray Group Management.


As emissions and radiances are adjusted, the groups are dynamically updated at predefined training intervals through the following process.
Within the uncertain group, we evaluate the expected emission strength of rays, retaining only those above a threshold k_i.
Rays below this threshold are then merged to the certain group:



    ℛ^U_i = {r | max_RGB(∫_0^∞ T(x)ρ(x) E(x)   dt) ≥ k_i, r ∈ℛ^U_i-1} ,


    ℛ^C_i = (ℛ^U_i-1 - ℛ^U_t) ∪ℛ^C_i-1 .






Subsequently, newly added rays to the certain group can be used to localize influential points and update their outgoing radiances.



This iterative process progressively refines the separation between reflective and emissive regions, attaining more accurate identification of emissive sources.




LTS Loss Decomposition.
The LTS loss, as detailed in Eq. <ref>, can be decomposed using a stop-gradient operation to refine the adjustment process.



    ℒ_lts^E = ∑_x,ω_o (    λ_l‖sg(L_o^E(x,ω_o)) - L̂_o^E(x,ω_o) ‖_1 + 
       λ_r‖ L_o^E(x,ω_o) - sg(L̂_o^E(x,ω_o)) ‖_1).






We prioritize λ_l to enhance the update of scene context, affecting other points' radiance given the predicted L_o(x,ω_o).
λ_r prevents severe deviation of every L_o(x,ω_o) within the current scene context.
This  aligns with our focus on HDR source reconstruction from LDR images, addressing under-represented information in training data.













































 §.§ Training Details











We employ the Voxurf architecture <cit.> as backbone and adopt the simplified Disney BRDF model <cit.> for SVBRDF representation, with parameters including base color ∈ [0, 1]^3, roughness ∈ [0, 1], and metallic ∈ [0, 1].
The learnable tone-mapper, structured as a two-layer MLP, is  utilized for the rendering loss only.



Initially, we pre-train our networks using the rendering loss, subsequently integrating the basic LTS loss (Eq. <ref> and Eq. <ref>) into our training regimen.
This phase transitions to the reflection-aware progressive training scheme, where we adopt the ℓ_1 loss due to its empirical stability in refining emissive source reconstruction.



We use a smoothing regularization to promote local consistency in normals, BRDFs, and emissions.
To ensure view-consistent labeling of 3D points as either reflective or emissive, we implement the emission suppression loss for points beloning to the certain group:




    ℒ_supp^E =∑_r∈ R_t^C‖∫_0^∞T(x)ρ(x)E(x)  dt ‖_2^2,






The threshold k_i linearly increases with each time step t, utilizing a grid search within a range of [10^-3, 10^-5] to find the slope.
We construct mini-batches via stratified sampling within each group.
For a detailed description of our training procedure, please refer to  Appendix.














































 §.§ Scene Editing






Reconstructed emissive sources enable scene editing;
users select emissive sources using binary masks M_j=1… N and specify lighting conditions using colors c_j=1… N and intensities i_j=1… N within the HSV color space <cit.>.




We identify the rays in the uncertain group that match M by projecting expected surface points p of the rays onto the camera with the pose 𝐑|𝐭:


    p = ∫_0^∞ T(x)ρ(x)x  dt,


    𝕀_j^hit(x)=interp(M_j,p')>0,  where  p'=𝐊[𝐑|𝐭][p|1]^T .






For the rays satisfying 𝕀_j^hit(x), we apply the designated lighting conditions.

The new emission values are computed by substituting the original hue (H) and saturation (S) of E(x) with the user-specified color c_j and adjusting the value (V) of v(x) with the new intensity i_j:



    E(x) = hsv_to_rgb([c_j | (v(x) × i)]) ·𝕀_j^hit + E(x) ·𝕀_j^hit .



These modifications influence scene appearance by optimizing the loss in Eq. <ref>.

During this process, all networks, except for L_o^E(x,ω_o), are frozen:


    ℒ_edit = ∑_x,ω_o‖ L_o^E(x,ω_o) - sg(L̂_o^E(x,ω_o)) ‖_2^2 .









§ EXPERIMENTS









We assess in reconstructing emissive sources by focusing on both identification and intensity restoration.

To showcase its effectiveness, we conduct a range of experiments, including scene editing, ablation studies, illumination decomposition, and surface reconstruction, providing both quantitative and qualitative results.





































 §.§ Experiment Settings


We curate 6 diverse synthetic scenes, each with 200 training images evenly distributed between on and off lighting conditions.



To evaluate the robustness of our approach against light colors, we consider two distinct settings of white colored and vivid colored emissive sources, resulting in a total of 12 scenes.  
The vivid colors are selected with full saturation in the HSV color space.


We measure source identification and radiance reconstruction using IoU and MSE metrics on novel view test images, comparing against ground truth data from Blender-rendered emission masks and EXR files.
The emission strengths, the maximum EXR file values,  range from 2 to 200.

For quantitative scene editing evaluation, we alter the white-colored sources to various colors—red, green, blue, cyan, magenta, yellow—and adjust intensities to half or double their original values.


Qualitative results include scene editing for vividly colored sources and real scenes captured with a Fuji 100s camera using Philips smart bulbs as emissive sources.
Quantitative assessments are based on 50 test images from novel camera poses, except for MSE measured for 25 test images.

We denote the best performance with blue and the second-best with green.


Additionally, we utilize the DTU dataset <cit.> to evaluate 's performance in surface reconstruction tasks where emissive sources are absent.














Baselines.
We select two state-of-the-art re-lighting methods, TensoIR <cit.> and NeILF++ <cit.>, that do not require prior lighting information.
For thorough evaluation, we also implement a simple method, Twins, where separate models are trained under light on and off conditions.
The Twins utilize the radiance discrepancies between the on and off models to distinguish and adjust emissive sources.


For scene editing, we add NeRF-W <cit.> and PaletteNeRF <cit.> as baselines.



Both NeRF-W and Twins adopt the Voxurf <cit.> architecture for fair comparison.

For methods unable to individually control emissive sources, all sources are adjusted together to match the last lighting condition by a user.

For the DTU dataset, we include state-of-the-art surface reconstruction methods that use object masks, such as NeuS <cit.> and Voxurf, as well as Neural-PBIR <cit.>, that jointly reconstructs surfaces, materials, and environment maps.








 §.§ Results


Emissive Source Recosntruction.
Tab. <ref> shows that our approach excels in accurately identifying emissive source regions and restoring their intensity, regardless of the source color.





















While TensoIR and NeILF++ can restore emissions by modifying their physical rendering equations, they suffer from emissive source ambiguity, leading to near-zero IoU performance (see Appendix).

For a comprehensive comparison, we report the best performance of the baseline methods using thresholding on the reconstructed emission strength at 0.01 intervals.

consistently outperforms the baselines in identifying emissive source regions across all scenes.

Our method also achieves significantly lower MSE values for restoring LDR to HDR images compared to the baselines, demonstrating its effectiveness of handling the ill-posed nature of the scenes with emissive sources.

This is visually confirmed in Fig. <ref>, where surpasses the baselines in a complex scene with numerous small light bulbs.



































Scene Editing.


Tab. <ref> and Fig. <ref> showcase the scene editing results under novel lighting conditions.
Baseline methods struggle to adapt to lighting changes due to their inability to reconstruct emissive sources accurately.


For example, in the Lego scene, TensoIR fails to adjust the illumination in surrounding regions when the color of emissive sources is changed, and in the Cube scene, both the hidden iPad screen and the cube surface covered by the user input mask change together.


Twins introduces blue light onto yellow and red surfaces, leading to unintended white and purple appearances, even though there should be no reflection.


PaletteNeRF, which manipulates scenes through re-colorization, lacks precise control over illumination, as seen in the synchronous color changes in the yellow ribbon and lighting.


In contrast, demonstrates superior performance in scene editing outshining all baselines thanks to the accurate identification of emissive sources, as detailed in Table <ref>.

effectively balances source reconstruction and novel view synthesis, ensuring high performance in both tasks.




NeRF-W is excluded from color adjustments since it doesn't support direct color change through interpolating latent variables learned with light on and off conditions.







Fig. <ref> to <ref> present additional examples of emissive source reconstruction and scene editing results.
Fig. <ref> shows results on real scenes, for which
due to the impracticality of precise control over smart bulb colors, we offer emission reconstruction results with pseudo ground truth data.

Our method effectively identifies emissive sources in real scenes, while it faces challenges in capturing complex reflections within light bulbs, as evident in the bright spot at the center of the bulbs in the ground truth edit results.















































Ablation Analysis.









Progressive refinement with the stop-gradient operation in Eq. <ref>  improves the identification of emissive sources and reduces MSE values.
Without m_θ, surface reconstructions become unreliable, complicating the accurate reconstruction of emissive sources.
This issue is evident from the CD metrics and illustrated in Fig. <ref>.
Further analyses are provided in  Appendix.























Illumination Decomposition.
Fig. <ref> demonstrates 's  decomposition of scene illumination into direct and indirect lighting from an environment map, as well as emissions and their reflections.

The shadow behind the yellow ribbon in the direct figure and the illumination in the indirect figure showcase 's ability to model both direct and indirect illumination.
The reflection figure shows that our method accurately captures how emissive sources contribute to reflections on nearby regions.








































































Surface Reconstruction.

Interestingly, our approach can be applied to the scenes without emissive sources to enhance surface reconstruction, as evidenced by the lower CD values in Tab. <ref> on the DTU dataset.



For this experiment, we use Eq. <ref> to <ref> without our progressive refinement technique.










's ability to adjust interrelated outgoing radiances helps  prevent surface formations where radiances cannot be produced, considering the predicted scene context.





Additional visualizations of the normals, BRDF, and environment maps are provided in  Appendix.




























































































































§ CONCLUSION


We present as the first NeRF-based inverse rendering method for the scenes with emissive sources.

Our approach uses LDR images, eliminating the need of HDR images to reconstruct emissive sources.



Furthermore, we demonstrate the application of reconstructed sources in scene editing, enabling color and intensity modifications.


Limitations.



















Future work could explore using a single lighting condition to disentangle emissive sources, environmental lighting, and object texture. 
It is also promising to address the challenge of volume ray tracing in unbounded scenes to extend to indoor scenes.



Additionally, LTS based re-lighting may be weak in representing new colors that traverse unobserved light paths during training. 



An alternative approach could be extracting emission texture maps and modifying it using the engines such as Blender <cit.> or Mitsuba <cit.>. 
More details on alternative re-lighting methods and radiance fine-tuning are provided in  Appendix.


§ ACKNOWLEDGEMENTS



This work was supported by Samsung Electronics MX, 
Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education(RS-2023-00274280),
and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01082, SW StarLab; No. 2022-0-00156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation).



Gunhee Kim is the corresponding author.



    102
        
        
        urlstyle

        [Attal et al.(2022)Attal, Huang, Zollhöfer, Kopf, and Kim]rayspace2022
        Benjamin Attal, Jia-Bin Huang, Michael Zollhöfer, Johannes Kopf, and Changil Kim.
        Learning neural light fields with ray-space embedding.
        In CVPR, 2022.

        [Azinovic et al.(2019)Azinovic, Li, Kaplanyan, and Nießner]inversepathtracing2019
        Dejan Azinovic, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner.
        Inverse path tracing for joint material and lighting estimation.
        In CVPR, 2019.

        [Barron et al.(2021)Barron, Mildenhall, Tancik, Hedman, Martin-Brualla, and Srinivasan]mipnerf2021
        Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.
        Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.
        In ICCV, 2021.

        [Barron et al.(2022)Barron, Mildenhall, Verbin, Srinivasan, and Hedman]360mipnerf2022
        Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.
        Mip-nerf 360: Unbounded anti-aliased neural radiance fields.
        In CVPR, 2022.

        [Barron et al.(2023)Barron, Mildenhall, Verbin, Srinivasan, and Hedman]barron2023zipnerf
        Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.
        Zip-nerf: Anti-aliased grid-based neural radiance fields.
        In ICCV, 2023.

        [Bi et al.(2020)Bi, Xu, Srinivasan, Mildenhall, Sunkavalli, Hašan, Hold-Geoffroy, Kriegman, and Ramamoorthi]reflectfields2020
        Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi.
        Neural reflectance fields for appearance acquisition.
        In arXiv, 2020.

        [Boss et al.(2020)Boss, Jampani, Kim, Lensch, and Kautz]twoshot2020
        Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz.
        Two-shot spatially-varying brdf and shape estimation.
        In CVPR, 2020.

        [Boss et al.(2021a)Boss, Braun, Jampani, Barron, Liu, and Lensch]nerd2021
        Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch.
        Nerd: Neural reflectance decomposition from image collections.
        In ICCV, 2021a.

        [Boss et al.(2021b)Boss, Jampani, Braun, Liu, Barron, and Lensch]neuralpil2021
        Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P.A. Lensch.
        Neural-pil: Neural pre-integrated lighting for reflectance decomposition.
        In NeurIPS, 2021b.

        [Burley and Studios(2012)]DisneyBRDF
        Brent Burley and Walt Disney Animation Studios.
        Physically-based shading at disney.
        In SIGGRAPH, 2012.

        [Cai et al.(2023)Cai, Huang, Jia, Lv, and Fu]cai2023neuda
        Bowen Cai, Jinchi Huang, Rongfei Jia, Chengfei Lv, and Huan Fu.
        Neuda: Neural deformable anchor for high-fidelity implicit surface reconstruction.
        In CVPR, 2023.

        [Chen et al.(2022)Chen, Xu, Geiger, Yu, and Su]tensorf2022
        Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.
        Tensorf: Tensorial radiance fields.
        In ECCV, 2022.

        [Community(2018)]blender
        Blender Online Community.
        Blender - a 3D modelling and rendering package.
        Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.

        [Fridovich-Keil et al.(2022)Fridovich-Keil, Yu, Tancik, Chen, Recht, and Kanazawa]plenoxels2021
        Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.
        Plenoxels: Radiance fields without neural networks.
        In CVPR, 2022.

        [Garon et al.(2019)Garon, Sunkavalli, Hadap, Carr, and Lalonde]Garon_2019_CVPR
        Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean-Francois Lalonde.
        Fast spatially-varying indoor lighting estimation.
        In CVPR, 2019.

        [Ge et al.(2023)Ge, Hu, Zhao, Liu, and Chen]ge2023ref
        Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen.
        Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection.
        In ICCV, 2023.

        [Hadadan et al.(2021)Hadadan, Chen, and Zwicker]radiosity2021
        Saeed Hadadan, Shuhong Chen, and Matthias Zwicker.
        Neural radiosity.
        In ACM TOG, 2021.

        [Hadadan et al.(2023)Hadadan, Lin, Novák, Rousselle, and Zwicker]hadadan2023inverse
        Saeed Hadadan, Geng Lin, Jan Novák, Fabrice Rousselle, and Matthias Zwicker.
        Inverse global illumination using a neural radiometric prior.
        In SIGGRAPH Conference Proceedings, 2023.

        [Haefner et al.(2021)Haefner, Green, Oursland, Andersen, Goesele, Cremers, Newcombe, and Whelan]Recovering2021
        Bjoern Haefner, Simon Green, Alan Oursland, Daniel Andersen, Michael Goesele, Daniel Cremers, Richard Newcombe, and Thomas Whelan.
        Recovering real-world reflectance properties and shading from hdr imagery.
        In 3DV, 2021.

        [Hasselgren et al.(2022)Hasselgren, Hofmann, and Munkberg]hasselgren2022nvdiffrecmc
        Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg.
        Shape, light, and material decomposition from images using monte carlo rendering and denoising.
        In NeurIPS, 2022.

        [IEC(1999)]iec61966-2-1
        IEC.
        IEC 61966-2-1:1999.
        Technical report, International Electrotechnical Commission, 1999.

        [Jakob et al.(2022)Jakob, Speierer, Roussel, Nimier-David, Vicini, Zeltner, Nicolet, Crespo, Leroy, and Zhang]Mitsuba3
        Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang.
        Mitsuba 3 renderer, 2022.
        https://mitsuba-renderer.org.

        [Jensen et al.(2014)Jensen, Dahl, Vogiatzis, Tola, and Aanæs]DTU
        Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs.
        Large scale multi-view stereopsis evaluation.
        In CVPR, 2014.

        [Ji et al.(2022)Ji, Yu, Guo, Liu, and Liu]ji2022relight
        Chaonan Ji, Tao Yu, Kaiwen Guo, Jingxin Liu, and Yebin Liu.
        Geometry-aware single-image full-body human relighting.
        In ECCV, 2022.

        [Jin et al.(2023)Jin, Liu, Xu, Zhang, Han, Bi, Zhou, Xu, and Su]Jin2023TensoIR
        Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su.
        Tensoir: Tensorial inverse rendering.
        In CVPR, 2023.

        [Kajiya(1986)]kajiya
        James T. Kajiya.
        The rendering equation.
        In SIGGRAPH, 1986.

        [Kellnhofer et al.(2021)Kellnhofer, Jebe, Jones, Spicer, Pulli, and Wetzstein]lumigraph2021
        Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein.
        Neural lumigraph rendering.
        In CVPR, 2021.

        [Kuang et al.(2023)Kuang, Luan, Bi, Shu, Wetzstein, and Sunkavalli]kuang2023palettenerf
        Zhengfei Kuang, Fujun Luan, Sai Bi, Zhixin Shu, Gordon Wetzstein, and Kalyan Sunkavalli.
        Palettenerf: Palette-based appearance editing of neural radiance fields.
        In CVPR, 2023.

        [Li et al.(2020)Li, Shafiei, Ramamoorthi, Sunkavalli, and Chandraker]li2020inverse
        Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker.
        Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image.
        In CVPR, 2020.

        [Li et al.(2022)Li, Shi, Bi, Zhu, Sunkavalli, Hašan, Xu, Ramamoorthi, and Chandraker]physically2022
        Zhengqin Li, Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Miloš Hašan, Zexiang Xu, Ravi Ramamoorthi, and Manmohan Chandraker.
        Physically-based editing of indoor scene lighting from a single image.
        In ECCV, 2022.

        [Li et al.(2023a)Li, Müller, Evans, Taylor, Unberath, Liu, and Lin]li2023neuralangelo
        Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin.
        Neuralangelo: High-fidelity neural surface reconstruction.
        In CVPR, 2023a.

        [Li et al.(2023b)Li, Wang, Cheng, Pan, and Yang]li2022texir
        Zhen Li, Lingli Wang, Mofang Cheng, Cihui Pan, and Jiaqi. Yang.
        Multi-view inverse rendering for large-scale real-world indoor scenes.
        In CVPR, 2023b.

        [Liang et al.(2023)Liang, Chen, Li, Chen, Panneer, and Vijaykumar]liang2023envidr
        Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar.
        Envidr: Implicit differentiable renderer with neural environment lighting.
        In ICCV, 2023.

        [Luan et al.(2021)Luan, Zhao, Bala, and Dong]luan2021unified
        Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong.
        Unified shape and svbrdf recovery using differentiable monte carlo rendering.
        In EGSR, 2021.

        [Lyu et al.(2022)Lyu, Tewari, Leimkuehler, Habermann, and Theobalt]lyu2022nrtf
        Linjie Lyu, Ayush Tewari, Thomas Leimkuehler, Marc Habermann, and Christian Theobalt.
        Neural radiance transfer fields for relightable novel-view synthesis with global illumination.
        In ECCV, 2022.

        [Lyu et al.(2023)Lyu, Tewari, Habermann, Saito, Zollhöfer, Leimküehler, and Theobalt]lyu2023dpi
        Linjie Lyu, Ayush Tewari, Marc Habermann, Shunsuke Saito, Michael Zollhöfer, Thomas Leimküehler, and Christian Theobalt.
        Diffusion posterior illumination for ambiguity-aware inverse rendering.
        In ACM TOG, 2023.

        [Mai et al.(2023)Mai, Verbin, Kuester, and Fridovich-Keil]mai2023neural
        Alexander Mai, Dor Verbin, Falko Kuester, and Sara Fridovich-Keil.
        Neural microfacet fields for inverse rendering.
        In ICCV, 2023.

        [Martin-Brualla et al.(2021)Martin-Brualla, Radwan, Sajjadi, Barron, Dosovitskiy, and Duckworth]nerfw2020
        Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.
        Nerf in the wild: Neural radiance fields for unconstrained photo collections.
        In CVPR, 2021.

        [Max(1995)]Max
        Nelson Max.
        Optical models for direct volume rendering.
        IEEE Transactions on Visualization and Computer Graphics, 10 (2):0 99–108, 1995.

        [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng]nerf2020
        Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.
        Nerf: Representing scenes as neural radiance fields for view synthesis.
        In ECCV, 2020.

        [Müller et al.(2022)Müller, Evans, Schied, and Keller]mueller2022instant
        Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.
        Instant neural graphics primitives with a multiresolution hash encoding.
        In ACM TOG, 2022.

        [Munkberg et al.(2022)Munkberg, Hasselgren, Shen, Gao, Chen, Evans, Müller, and Fidler]Munkberg_2022_CVPR
        Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler.
        Extracting triangular 3d models, materials, and lighting from images.
        In CVPR, 2022.

        [Niemeyer et al.(2020)Niemeyer, Mescheder, Oechsle, and Geiger]DVR
        Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.
        Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.
        In CVPR, 2020.

        [Oechsle et al.(2021)Oechsle, Peng, and Geiger]Oechsle2021ICCV
        Michael Oechsle, Songyou Peng, and Andreas Geiger.
        Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.
        In ICCV, 2021.

        [Ost et al.(2022)Ost, Laradji, Newell, Bahat, and Heide]pointlight2022
        Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide.
        Neural point light fields.
        In CVPR, 2022.

        [Pandey et al.(2021)Pandey, Orts-Escolano, LeGendre, Haene, Bouaziz, Rhemann, Debevec, and Fanello]totalrelighting2021
        Rohit Pandey, Sergio Orts-Escolano, Chloe LeGendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Seann Fanello.
        Total relighting: Learning to relight portraits for background replacement.
        In ACM TOG, 2021.

        [Park et al.(2019)Park, Florence, Straub, Newcombe, and Lovegrove]Park_2019_CVPR
        Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.
        Deepsdf: Learning continuous signed distance functions for shape representation.
        In CVPR, 2019.

        [Philip et al.(2021)Philip, Morgenthaler, Gharbi, and Drettakis]freeviewpoint2021
        Julien Philip, Sébastien Morgenthaler, Michaël Gharbi, and George Drettakis.
        Free-viewpoint indoor neural relighting from multi-view stereo.
        In ACM TOG, 2021.

        [Piala and Clark(2021)]terminerf2021
        Martin Piala and Ronald Clark.
        Terminerf: Ray termination prediction for efficient neural rendering.
        In 3DV, 2021.

        [Rudnev et al.(2022)Rudnev, Elgharib, Smith, Liu, Golyanik, and Theobalt]nerfosr2022
        Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt.
        Nerf for outdoor scene relighting.
        In ECCV, 2022.

        [Schonberger and Frahm(2016)]Schonberger_2016_CVPR
        Johannes L. Schonberger and Jan-Michael Frahm.
        Structure-from-motion revisited.
        In CVPR, 2016.

        [Sitzmann et al.(2020)Sitzmann, Martel, Bergman, Lindell, and Wetzstein]sitzmann2019siren
        Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.
        Implicit neural representations with periodic activation functions.
        In NeurIPS, 2020.

        [Sitzmann et al.(2021)Sitzmann, Rezchikov, Freeman, Tenenbaum, and Durand]lightfield2021
        Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand.
        Light field networks: Neural scene representations with single-evaluation rendering.
        In NeurIPS, 2021.

        [Smith(1978)]HSV
        Alvy Ray Smith.
        Color gamut transform pairs.
        In ACM TOG, 1978.

        [Srinivasan et al.(2021)Srinivasan, Deng, Zhang, Tancik, Mildenhall, and Barron]nerv2021
        Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron.
        Nerv: Neural reflectance and visibility fields for relighting and view synthesis.
        In CVPR, 2021.

        [Srinivasan et al.(2023)Srinivasan, Garbin, Verbin, Barron, and Mildenhall]srinivasan2023nuvo
        Pratul P. Srinivasan, Stephan J. Garbin, Dor Verbin, Jonathan T. Barron, and Ben Mildenhall.
        Nuvo: Neural uv mapping for unruly 3d representations.
        arXiv, 2023.

        [Suhail et al.(2022)Suhail, Esteves, Sigal, and Makadia]lightfield2022
        Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia.
        Light field neural rendering.
        In CVPR, 2022.

        [Sun et al.(2022a)Sun, Sun, and Chen]dvgo2022
        Cheng Sun, Min Sun, and Hwann-Tzong Chen.
        Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.
        In CVPR, 2022a.

        [Sun et al.(2023)Sun, Cai, Li, Yan, Zhang, Marshall, Huang, Zhao, and Dong]sun2023neuralpbir
        Cheng Sun, Guangyan Cai, Zhengqin Li, Kai Yan, Cheng Zhang, Carl Marshall, Jia-Bin Huang, Shuang Zhao, and Zhao Dong.
        Neural-pbir reconstruction of shape, material, and illumination.
        In ICCV, 2023.

        [Sun et al.(2022b)Sun, Chen, Wang, Li, Averbuch-Elor, Zhou, and Snavely]sun2022neuconw
        Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, and Noah Snavely.
        Neural 3D reconstruction in the wild.
        In SIGGRAPH Conference Proceedings, 2022b.

        [Sun et al.(2021)Sun, Lin, Bi, Xu, and Ramamoorthi]nelf2021
        Tiancheng Sun, Kai-En Lin, Sai Bi, Zexiang Xu, and Ravi Ramamoorthi.
        Nelf: Neural light-transport field for portrait view synthesis and relighting.
        In EGSR, 2021.

        [Tancik et al.(2020)Tancik, Srinivasan, Mildenhall, Fridovich-Keil, Raghavan, Singhal, Ramamoorthi, Barron, and Ng]tancik2020fourfeat
        Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng.
        Fourier features let networks learn high frequency functions in low dimensional domains.
        In NeurIPS, 2020.

        [Tang et al.(2023)Tang, Zhou, Chen, Hu, Ding, Wang, and Zeng]tang2022nerf2mesh
        Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng.
        Delicate textured mesh recovery from nerf via adaptive surface refinement.
        In ICCV, 2023.

        [Tewari et al.(2020)Tewari, Fried, Thies, Sitzmann, Lombardi, Sunkavalli, Martin-Brualla, Simon, Saragih, Nießner, et al.]sotarendering2020
        Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al.
        State of the art on neural rendering.
        In CGF, 2020.

        [Tewari et al.(2022)Tewari, Thies, Mildenhall, Srinivasan, Tretschk, Yifan, Lassner, Sitzmann, Martin-Brualla, Lombardi, et al.]advancesrendering2022
        Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al.
        Advances in neural rendering.
        In CGF, 2022.

        [Toschi et al.(2023)Toschi, De Matteo, Spezialetti, De Gregorio, Di Stefano, and Salti]Toschi_2023_CVPR
        Marco Toschi, Riccardo De Matteo, Riccardo Spezialetti, Daniele De Gregorio, Luigi Di Stefano, and Samuele Salti.
        Relight my nerf: A dataset for novel view synthesis and relighting of real world objects.
        In CVPR, 2023.

        [Verbin et al.(2022)Verbin, Hedman, Mildenhall, Zickler, Barron, and Srinivasan]refnerf2022
        Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan.
        Ref-nerf: Structured view-dependent appearance for neural radiance fields.
        In CVPR, 2022.

        [Walter et al.(2007)Walter, Marschner, Li, and Torrance]walter
        Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance.
        Microfacet models for refraction through rough surfaces.
        In Proceedings of the 18th Eurographics Conference on Rendering Techniques, 2007.

        [Wang et al.(2023a)Wang, Zhang, and Süsstrunk]nemto
        Dongqing Wang, Tong Zhang, and Sabine Süsstrunk.
        Nemto: Neural environment matting for novel view and relighting synthesis of transparent objects.
        In ICCV, 2023a.

        [Wang et al.(2009)Wang, Ren, Gong, Snyder, and Guo]SG
        Jiaping Wang, Peiran Ren, Minmin Gong, John Snyder, and Baining Guo.
        All-frequency rendering of dynamic, spatially-varying reflectance.
        ACM TOG, 2009.

        [Wang et al.(2021a)Wang, Liu, Liu, Theobalt, Komura, and Wang]wang2021neus
        Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.
        Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.
        In NeurIPS, 2021a.

        [Wang et al.(2022a)Wang, Schwarz, Geiger, and Tang]ARAH2022
        Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang.
        Arah: Animatable volume rendering of articulated human sdfs.
        In ECCV, 2022a.

        [Wang et al.(2023b)Wang, Han, Habermann, Daniilidis, Theobalt, and Liu]neus2
        Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu.
        Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction.
        In ICCV, 2023b.

        [Wang et al.(2021b)Wang, Philion, Fidler, and Kautz]wang2021learning
        Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz.
        Learning indoor inverse rendering with 3d spatially-varying lighting.
        In ICCV, 2021b.

        [Wang et al.(2022b)Wang, Chen, Acuna, Kautz, and Fidler]wang2022neural
        Zian Wang, Wenzheng Chen, David Acuna, Jan Kautz, and Sanja Fidler.
        Neural light field estimation for street scenes with differentiable virtual object insertion.
        In ECCV, 2022b.

        [Wang et al.(2023c)Wang, Shen, Gao, Huang, Munkberg, Hasselgren, Gojcic, Chen, and Fidler]wang2023fegr
        Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, and Sanja Fidler.
        Neural fields meet explicit geometric representations for inverse rendering of urban scenes.
        In CVPR, 2023c.

        [Weber et al.(2022)Weber, Garon, and Lalonde]hweberEditableIndoorLight
        Henrique Weber, Mathieu Garon, and Jean-François Lalonde.
        Editable indoor lighting estimation.
        In ECCV, 2022.

        [Wu et al.(2023a)Wu, Hu, Li, Zhang, Fan, and Yu]wu2023nefii
        Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, and Xin Yu.
        Nefii: Inverse rendering for reflectance decomposition with near-field indirect illumination.
        In CVPR, 2023a.

        [Wu et al.(2023b)Wu, Zhu, Yaldiz, Zhu, Cai, Matai, Porikli, Li, Chandraker, and Ramamoorthi]fipt2023
        Liwen Wu, Rui Zhu, Mustafa B. Yaldiz, Yinhao Zhu, Hong Cai, Janarbek Matai, Fatih Porikli, Tzu-Mao Li, Manmohan Chandraker, and Ravi Ramamoorthi.
        Factorized inverse path tracing for efficient and accurate material-lighting estimation.
        In ICCV, 2023b.

        [Wu et al.(2022)Wu, Liu, Chen, Li, Zheng, Cai, and Zheng]wu2022object
        Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng.
        Object-compositional neural implicit surfaces.
        In ECCV, 2022.

        [Wu et al.(2023c)Wu, Wang, Li, Zheng, and Cai]wu2023objsdfplus
        Qianyi Wu, Kaisiyuan Wang, Kejie Li, Jianmin Zheng, and Jianfei Cai.
        Objectsdf++: Improved object-compositional neural implicit surfaces.
        In ICCV, 2023c.

        [Wu et al.(2023d)Wu, Sun, Lai, and Gao]DE-NeRF
        Tong Wu, Jia-Mu Sun, Yu-Kun Lai, and Lin Gao.
        De-nerf: Decoupled neural radiance fields for view-consistent appearance editing and high-frequency environmental relighting.
        In SIGGRAPH ASIA Conference Proceedings, 2023d.

        [Wu et al.(2023e)Wu, Wang, Pan, Xu, Theobalt, Liu, and Lin]wu2022voxurf
        Tong Wu, Jiaqi Wang, Xingang Pan, Xudong Xu, Christian Theobalt, Ziwei Liu, and Dahua Lin.
        Voxurf: Voxel-based efficient and accurate neural surface reconstruction.
        In ICLR, 2023e.

        [Xiangli et al.(2022)Xiangli, Xu, Pan, Zhao, Rao, Theobalt, Dai, and Lin]xiangli2022bungeenerf
        Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin.
        Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering.
        In ECCV, 2022.

        [Xu et al.(2023)Xu, Zoss, Chandran, Gross, Bradley, and Gotardo]xu2023renerf
        Yingyan Xu, Gaspard Zoss, Prashanth Chandran, Markus Gross, Derek Bradley, and Paulo Gotardo.
        Renerf: Relightable neural radiance fields with nearfield lighting.
        In ICCV, 2023.

        [Yang et al.(2022a)Yang, Chen, Chen, Chen, and Wong]psnerf2022
        Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan-Yee K. Wong.
        Ps-nerf: Neural inverse rendering for multi-view photometric stereo.
        In ECCV, 2022a.

        [Yang et al.(2022b)Yang, Chen, Chen, Chen, and Wong]yang2022s3nerf
        Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan-Yee K. Wong.
        S^3-nerf: Neural reflectance field from shading and shadow under a single viewpoint.
        In NeurIPS, 2022b.

        [Yao et al.(2022)Yao, Zhang, Liu, Qu, Fang, McKinnon, Tsin, and Quan]neural-incident-light-field
        Yao Yao, Jingyang Zhang, Jingbo Liu, Yihang Qu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan.
        Neilf: Neural incident light field for material and lighting estimation.
        In ECCV, 2022.

        [Yariv et al.(2020)Yariv, Kasten, Moran, Galun, Atzmon, Ronen, and Lipman]idr2020
        Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman.
        Multiview neural surface reconstruction by disentangling geometry and appearance.
        In NeurIPS, 2020.

        [Yariv et al.(2021)Yariv, Gu, Kasten, and Lipman]volsdf2021
        Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.
        Volume rendering of neural implicit surfaces.
        In NeurIPS, 2021.

        [Yu et al.(2021)Yu, Li, Tancik, Li, Ng, and Kanazawa]yu2021plenoctrees
        Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.
        PlenOctrees for real-time rendering of neural radiance fields.
        In ICCV, 2021.

        [Zeng et al.(2023)Zeng, Chen, Dong, Peers, Wu, and Tong]zeng2023nrhints
        Chong Zeng, Guojun Chen, Yue Dong, Pieter Peers, Hongzhi Wu, and Xin Tong.
        Relighting neural radiance fields with shadow and highlight hints.
        In SIGGRAPH Conference Proceedings, 2023.

        [Zhang et al.(2023a)Zhang, Yao, Li, Liu, Fang, McKinnon, Tsin, and Quan]zhang2023neilf++
        Jingyang Zhang, Yao Yao, Shiwei Li, Jingbo Liu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan.
        Neilf++: Inter-reflectable light fields for geometry and material estimation.
        In ICCV, 2023a.

        [Zhang et al.(2020)Zhang, Riegler, Snavely, and Koltun]nerf++2020
        Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.
        Nerf++: Analyzing and improving neural radiance fields.
        arXiv preprint arXiv:2010.07492, 2020.

        [Zhang et al.(2021a)Zhang, Luan, Wang, Bala, and Snavely]physg2021
        Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely.
        Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting.
        In CVPR, 2021a.

        [Zhang et al.(2022a)Zhang, Luan, Li, and Snavely]iron-2022
        Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely.
        Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images.
        In CVPR, 2022a.

        [Zhang et al.(2021b)Zhang, Fanello, Tsai, Sun, Xue, Pandey, Orts-Escolano, Davidson, Rhemann, Debevec, et al.]lighttransport2021
        Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, et al.
        Neural light transport for relighting and view synthesis.
        In ACM TOG, 2021b.

        [Zhang et al.(2021c)Zhang, Srinivasan, Deng, Debevec, Freeman, and Barron]nerfactor2021
        Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron.
        Nerfactor: Neural factorization of shape and reflectance under an unknown illumination.
        In ACM TOG, 2021c.

        [Zhang et al.(2022b)Zhang, Sun, He, Fu, Jia, and Zhou]modelingindirect2022
        Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou.
        Modeling indirect illumination for inverse rendering.
        In CVPR, 2022b.

        [Zhang et al.(2023b)Zhang, Hu, Wu, Zhao, Li, Zou, and Fan]zhang2023unbiased
        Y. Zhang, Z. Hu, H. Wu, M. Zhao, L. Li, Z. Zou, and C. Fan.
        Towards unbiased volume rendering of neural implicit surfaces with geometry priors.
        In CVPR, 2023b.

        [Zheng et al.(2021)Zheng, Singh, and Seidel]media2021
        Quan Zheng, Gurprit Singh, and Hans-Peter Seidel.
        Neural relightable participating media rendering.
        In NeurIPS, 2021.

        [Zhu et al.(2022)Zhu, Luan, Huo, Lin, Zhong, Xi, Wang, Bao, Zheng, and Tang]zhu2022learning
        Jingsen Zhu, Fujun Luan, Yuchi Huo, Zihao Lin, Zhihua Zhong, Dianbing Xi, Rui Wang, Hujun Bao, Jiaxiang Zheng, and Rui Tang.
        Learning-based inverse rendering of complex indoor scenes with differentiable monte carlo raytracing.
        In SIGGRAPH ASIA Conference Proceedings, 2022.

    







-1



§ APPENDIX




 §.§ Implementation Details


Training Procedure.
Our implementation builds upon Voxurf <cit.>, excluding its dual-color network feature.
We adhere to the coarse and fine processing stages described in Voxurf before initiating our LTS learning-based training strategy.
Additionally, we compute ray colors using alpha masks to filter out points in empty space, aligning with practices in previous studies <cit.>.
The LTS learning training procedure with progressive refinement approach is:


    
  * Initialize ray groups: Uncertain rays R^U_0=R and certain rays R^C_0=∅.
    
  * Form mini-batches using stratified sampling within each ray group.
    
  * Calculate the rendering loss, ℒ_render.
    
  * For rays in the mini-batch, uniformly sample 100 points to evaluate the LTS loss, ℒ_lts.
    
  * Compute the surface normal at sampled points.
    
  * For L_o^E(x,ω_o), sample an additional viewing direction on the upper hemisphere at these points.
    
  * For L̂_o^E(x,ω_o), sample 256 rays on the upper hemisphere at these points to compute incident radiance.
    
  * Calculate ℒ_lts, considering the group membership of each point.
    
  * Update network parameters.
    
  * Adjust ray groups at specified training intervals.
    
  * Repeat steps 2 through 10 until training ends.





Discretization.

Following NeuS <cit.>, we approximate ray color computation using N discrete points sampled along the ray, denoted as {x_i=c-t_iω_o | i=1,...,N, t_i<t_i+1}:

    Ĉ(r)=∑_i=1^NT_iα_iL_o(x_i,ω_o),


    α_i = max(Φ_s(f(x_i)) - Φ_s(f(x_i+1))/Φ_s(f(x_i)), 0),


    T_i = ∏_j=1^i-1(1-α_j).

α is the discrete equivalent of the SDF-based opacity, ρ.

For reflections in L̂_o(x,ω), we employ Monte Carlo sampling, uniformly sampling directions ω_i around the normal n at point x on the upper hemisphere.
While the current implementation of doesn't include importance sampling for incident rays, incorporating it in future work for variance reduction may enhance overall performance.

    L̂_o(x,ω_o)=E(x)+1/M∑_j=1^M(L_i(x,ω_j) R(x,ω_o,ω_j)/1/2π).




Simplified Diseny BRDF.
We adopt the simplified Disney principled BRDF function <cit.>, parameterized by base color b, metallic m, and roughness r.

    R(x,ω_o,ω_i)     = D(h,n,r)F(ω_o,h,b,m)G(ω_o,ω_i,h,r)/4(n·ω_o)
        + (n·ω_i)(1 - m) (b/π) ,

The half vector h is defined as h=ω_o + ω_i/‖ω_o + ω_i ‖_2.
Following NeILF++ <cit.>, the normal distribution function D is approximated using Spherical Gaussian:

    D(h,n,r) = 1/π r^4exp(2/r^4(h· n - 1)),

The Fresnel term F is calculated as follows:

    F(ω_o,h,b,m)     =F_0 + (1-F_0)(1-(ω_o · h)^5), 
    where F_0      =0.04(1-m) + bm,

The geometry term G adopts the GGX function <cit.>.

    G(ω_o, ω_i,n,r)     = (n·ω_o)(n ·ω_i)/( (n·ω_o)(1-k) + k ) ((n·ω_i)(1-k)+k), 
    where  k               = r^2/2.

For simplicity, our BRDF model incorporates the Lambert cosine term (n·ω_i).



Gamma Correction.
To ensure HDR linear color space for outgoing radiance, we apply the standard gamma correction as defined by IEC <cit.> to ray colors before calculating the rendering loss.
The gamma-corrected sRGB color, given a linear color C_linear, is computed as follows:

    τ(C_linear) =
        
            12.92C_linear   if  C_linear≤ 0.0031308, 
    
            1.055C_linear^1/2.4 - 0.055    if  C_linear > 0.0031308.




RGB to HSV.
For scene editing tasks, we utilize the HSV color model <cit.>.
The hue (H∈[0,1]), saturation (S∈[0,1]), and value (V∈ℝ_+) are calculated using the following method:


    M     = max(R,G,B), 
    
            m     = min(R,G,B), 
    
            C     = M - m.



    H      = (H' / 6.0)  1.0, 
    
            H'     =
            
                0                  if  C = 0, 
    G - B/C   if  M=R,   
    B-R/C+2    if  M=G,   
    R-G/C+4    if  M=B.



    S =
        
            0              if  V = 0, 
    C/V   otherwise,



    V = max(R,G,B)





HSV to RGB.
Once the color is replaced and intensity is adjusted in the HSV space, the conversion back to RGB is performed as:

    m              = V - C,                                    
    
            H'             = H × 6.0,                             
    
            C              = S× V,                                
    
            X              = C× (1 - | H' 2 - 1 |), 
    
            (R',G',B')     =
            
                (C, X, 0)    if  0≤ H' < 1, 
    
                (X, C, 0)    if  1≤ H' < 2, 
    
                (0, C, X)    if  2≤ H' < 3, 
    
                (0, X, C)    if  3≤ H' < 4, 
    
                (X, 0, C)    if  4≤ H' < 5, 
    
                (C, 0, X)    if  5≤ H' < 6.



    (R,G,B) =(R'+m,G'+m,B'+m) .




 §.§ Dataset Details


Dataset Construction.
This section outlines the dataset used for training and evaluation.
Each scene in our dataset comprises 200 training images, with an equal split between two lighting conditions: “on” and “off”.
Emission masks are utilized as ground truth for emissive source identification, while EXR files with linear pixel values assess the accuracy of the reconstructed strength of emission and reflection.

All data are rendered using the Cycles path tracing in Blender <cit.>, with settings that could artificially alter scene illumination are disabled, such as incdient light clamping and the Filmic transform.
For scene editing under novel lighting conditions, we introduce a variety of test scenarios, including intensity editing, color editing, and combined intensity and color editing, each with 50 images.
We derive these scenarios from 25 unique camera positions from the novel view evaluation dataset, each under two different lighting conditions,
Intensity adjustments are made relative to the original scene's emissive source strength, with “0” indicating “light off” and “1” matching the “light on” intensity.
We test intensity adjustments at half (0.5) and double (2.0) the original levels.

In scenes allowing individual source adjustments, we include an additional intensity condition where lights are selectively turned off (0.0).
For color editing, we select six colors—red, green, blue, cyan, magenta, and yellow—to demonstrate the effects of various light source colors on scene illumination.




Scene Characteristics.
Our scenes are meticulously crafted using assets from Blendswap and cgtrader, with licensing details and the count of emissive sources detailed in Tab. <ref>.
Below, we describe the unique aspects of each scene.





    
  * LEGO:
          This scene showcases three emissive sources, all starting with the same color and intensity.
          The intricate designs of the LEGO bricks create complex reflection effects.
          
          The emissive sources in these scenes are tested for both collective and individual adjustments.

          

    
  * Gift:
          Featuring a gift box, a toy, and numerous small bulbs, this scene presents a challenge with its multitude of tiny light bulbs and extensive reflection areas.

          

    
  * Book:
          The Book scene features a single large light source consisting of a lamp, a book, and a pencil.
          The emphasis here is on identifying and restoring the very large emissive source.

          

    
  * Cube:
          Comprising a tablet PC and a cube, this scene is marked by its sophisticated reflection effects, especially on the cube surfaces which varying albedo.

          

    
  * Billboard:
          This scene includes two billboards, each equipped with three emissive sources, summing up to six sources.
          The lights are positioned to shine downwards from the billboards' tops.
          We adjust the emissive sources collectively and individually.
          Individual adjustments are performed for three light groups by pairing the light sources of the front and back billboards.

          

    
  * Balls:
          This is the material balls scene in NeRF, with the modification of the red ball as an emissive source.

          
























































 §.§ Baseline Implementation


In our evaluation, we compared against two leading re-lighting methods, TensoIR <cit.> and NeILF++ <cit.>, known for their ability to operate without prior knowledge of scene components
Additionally, we made Twins, a method focused on emissive source reconstruction without relying on inverse rendering techniques.
For an in-depth analysis of scene editing capabilities, we include PaletteNeRF <cit.>, which achieves scene modification through re-colorization, and NeRF-W <cit.>, which adjusts scene illumination by interpolating between learned latent vectors.
For surface reconstruction evaluations on the DTU dataset, we selected state-of-the-art methods such as Voxurf <cit.> and NeuS <cit.>, alongside Neural-PBIR <cit.>, which offers a joint reconstruction of surfaces, materials, and environment maps.
We utilized the official implementations provided by the authors for all baselines, with the exception of NeRF-W.
We used the official implementation codes provided by the authors for all baseline methods except for Twins and NeRF-W.
Twins employs a dual-model strategy for 'light-on' and 'light-off' conditions, using radiance differences for emissive source identification and scene illumination editing.
NeRF-W leverages two latent embeddings for similar purposes, focusing on intensity adjustments.
Both Twins and NeRF-W are based on the Voxurf architecture to ensure a fair comparison with .
For NeILF++, we omitted the use of prior scene information to align with methods that do not use geometry hints like object meshes or oriented point clouds.
Neural-PBIR was excluded from emissive source reconstruction experiments as the code is not publicly available yet.
Baseline performance data on the DTU dataset are borrowed directly from the Voxurf, NeuS, and Neural-PBIR papers.





 §.§ Real Scene












We showcase the effectiveness of in identifying emissive sources in real-world scenes.
Camera poses are estimated using COLMAP <cit.>.
We use commercial smart light bulbs from Philips, which offer control over light colors.
Since precise control over the color of the smart bulbs is infeasible, we provide qualitative results for emissive source identification and scene editing in real scenes.
Fig. <ref> presents the decomposed scene components, such as normal, base color, roughness, metallic, and the environment map.
In Fig. <ref>, our method successfully identifies emissive sources, enabling scene illumination adjustments.
Fig. <ref> presents qualitative results for comparison with ground truth data.


Although our model successfully identifies emissive sources, it encounters difficulties with complex reflections inside light bulbs, as indicated by the bright spots at the bulb centers in the ground truth edit images.
Despite these challenges, stands out as the first NeRF-based inverse rendering method to address the reconstruction of emissive sources, enabling scene illumination modifications through the identification of light sources within a scene.

































 §.§ Reconstructed Scene Components

We present the reconstructed components of our synthetic scenes, including emissions, surface normals, and BRDF, in Fig. <ref> and <ref>.
We also provide the comparison of the reconstructed emission and BRDF performance among TensoIR, NeILF++, and in Fig. <ref> and <ref> for real scenes and Fig. <ref> to <ref> for synthetic scenes.

TensoIR and NeILF++ encounter difficulties, as does , in capturing precise roughness, often resulting in shadows being baked into the albedo.
This issue is exacerbated by a relatively dark environment map, in contrast to previous works, and is compounded by strong emissions and shadows.
Nevertheless, while BRDF results are comparable, distinguishes itself in its primary goal: the accurate reconstruction of emissive sources
We also provide the reconstructed scene components on DTU dataset in Fig. <ref> and <ref>.
































































 §.§ Illumination Decomposition

We present additional results of decomposed illumination in Fig. <ref>.
These visualizations offer insights into the effectiveness of in factorizing the scene illumination.
The off image, for instance, is generated by merging direct and indirect illumination from the environment map, as shown in the first row.
The second row illustrates the decomposition of emission effects, including both the emission and its reflection.
Light-on images are created by adding the light-off and the emission effects images.



 §.§ Scene Editing w&w/o Radiance Fine-tuning

Fig. <ref> and <ref> present additional scene editing examples, illustrating various scenarios including intensity and color edits, as well as their combination.
As discussed in the conclusion section of the main paper, scene illumination can be adjusted without fine-tuning radiance fields, using alternative methods.
Results on the right side of Fig. <ref> to <ref> are rendered by calculating only direct illumination from emissive sources for re-lighting, a technique commonly used in prior research <cit.>, bypassing the fine-tuning of trained networks.
This approach is particularly effective for scenes with vividly colored emissive sources, as shown in Fig. <ref>.
To evaluate the effectiveness of direct illumination in scene editing, we provide quantitative results for each scenario in Tab. <ref> and <ref>.
Quantitative comparisons for scenes with vivid-colored emissive sources are detailed in Tab. <ref> and <ref>.

















































 §.§ Analysis of Learnable Tone-mapper


We eliminate the constraint on the range of radiance values to address the unbounded nature of emissive sources and their reflections.
Instead of the commonly used sigmoid activation function in NeRF-based methods <cit.> for radiance prediction, we employ the softplus activation, extending the radiance range from [0,1] to [0,∞].

However, this modification may lead to inaccurate surface reconstructions, as highlighted in the main paper.
Fig. <ref> shows instances where surfaces become semi-transparen, lose structural details, and the rendered images significantly deviate from the ground truth, making the accurate reconstruction of emissive sources infeasible.

To address this issue, we introduce a learnable tone-mapper m_θ, taking positionally encoded HDR linear color as input and produce LDR sRGB colors outputs.
Fig. <ref> reveals that this tone-mapper helps in obtaining accurate surface normals and rendering photo-realistic images.
Nonetheless, a trade-off exists between the quality of surface normals and rendered images, when using the learnable tone-mapper.
For example, a low λ_τ value, which indicates a heavier reliance on the tone-mapper in the rendering loss, may improve surface details but linear color values deviate significantly from expectations.
This discrepancy occurs as the correlation between predicted linear colors and actual image pixel colors weakens with lower λ_τ values.
Conversely, a higher λ_τ compromises surface reconstruction quality.
Thus, setting λ_τ requires careful consideration of the balance between surface detail and color accuracy.

Interestingly, the choice of λ_τ also impacts the reconstruction of emissive sources in real scenes.
A high λ_τ tends to result in lower intensity of reconstructed emissive sources.
Re-lighting experiments in Fig. <ref> show illumination effects confined to a narrow area compared to ground truth data.
We suspect the camera may edit images for low contrast and apply color grading, particularly in HDR scenes.
We used the Fuji 100s camera.
A high λ_τ in the rendering loss could be problematic, as it aims to align gamma-corrected linear values with manipulated colors.
Based on this insight, we slightly reduced λ_τ by 0.1 to enhance emission intensity (1.4 vs. 37.2) and expand reflections in re-lighting scenarios.





























 §.§ Near-zero IoU Results of Baselines

State-of-the-art re-lighting methods struggle with ambiguities surrounding emissive sources, often failing to accurately identify them.
These methods typically cannot differentiate between reflections and emissions, leading to most regions being misclassified as emissive sources.
This challenge is reflected in Tab. <ref>, where baseline methods exhibit near-zero IoU performance across various scenes.
Despite extensive trinary grid searches with an interval of 0.01 for thresholding values to report the peak performance of baselines, consistently outperforms them.
Additionally, our method's efficacy in classifying rays into the uncertain group for emissive source identification highlights its superiority in this task.
This is further supported by additional results obtained using thresholding techniques applied to the baselines.






 §.§ Failure Cases in Scene Editing

We also present failure cases in scene editing, discussing the limitations of the radiance fine-tuning method for re-lighting in 4.5 of the main paper.
While effectively reconstructs and manipulates emissive sources, the radiance fine-tuning method for re-lighting has its limitations.
These are depicted in Fig. <ref>, where we note that LTS learning-based radiance fine-tuning may be constrained to color adjustments within the training spectrum.
In other words, using the LTS loss to transfer radiance within light transport segments may be weak in representing new colors that traverse unobserved light paths during training.
For example, it can shift colors from yellow to green but not to blue.
Additionally, the network's inherent smoothness capability may introduce illumination inaccuracies.
In the last row in Fig. <ref>, changing only the top emissive source to red inadvertently affects the bulldozer's lower ceiling.

Exploring alternative rendering approaches could address these issues.
We showcase scene editing results by computing direct illumination from emissive sources in Fig. <ref>, enabling changes to any colors.
Reconstructing emissive sources using , then extracting emission texture maps to use rendering engines like Blender <cit.> or Mitsuba <cit.> is also promising.
Howver, the texture map extraction in NeRF-like methods often faces severe UV atlas fragmentation.
Recent methods like Nuvo <cit.> offer some hope for feasible emission texture editing.
We consider these avenues for future exploration


































































































































































































































































































































































































