Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion
Yu-Jie Liang Zihan Cao Liang-Jian Deng Xiao Wu

Impact of Ricci Inverse Gravity on Hybrid Star Model
    Ayesha Almas
    April 29, 2024
====================================================
















Recently, implicit neural representations (INR) have made significant strides in various vision-related domains, providing a novel solution for Multispectral and Hyperspectral Image Fusion (MHIF) tasks. However, INR is prone to losing high-frequency information and is confined to the lack of global perceptual capabilities. To address these issues, this paper introduces a Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for MHIF task, targeting the following phenomena: The Fourier amplitudes of the HR-HSI latent code and LR-HSI are remarkably similar; however, their phases exhibit different patterns. In FeINFN, we innovatively propose a spatial and frequency implicit fusion function (Spa-Fre IFF), helping INR capture high-frequency information and expanding the receptive field. Besides, a new decoder employing a complex Gabor wavelet activation function, called Spatial-Frequency Interactive Decoder (SFID), is invented to enhance the interaction of INR features. Especially, we further theoretically prove that the Gabor wavelet activation possesses a time-frequency tightness property that favors learning the optimal bandwidths in the decoder. Experiments on two benchmark MHIF datasets verify the state-of-the-art (SOTA) performance of the proposed method, both visually and quantitatively. Also, ablation studies demonstrate the mentioned contributions. The code will be available on https://anonymous.4open.science/r/FeINFN-15C9/Anonymous GitHub after possible acceptance.




§ INTRODUCTION



Hyperspectral imaging captures scenes across contiguous spectral bands, offering intricate details compared to traditional single or limited-band images, and 
improving computer vision application accuracy, such as target recognition, classification, tracking, and segmentation <cit.>. However, practical optical sensors face challenges in balancing spatial resolution and spectral precision. Images with over 100 bands often exhibit lower spatial resolution, while those with fewer bands display higher spatial resolution.
Efforts for MHIF are underway to fuse high spatial-resolution multispectral images (HR-MSI) with low spatial-resolution hyperspectral images (LR-HSI) to finally obtain high spatial-resolution hyperspectral images (HR-HSI). Actually, MHIF technology could fuse hyperspectral images with multispectral images, extracting information not detectable by HR-MSI to enhance richness and precision. Recent MHIF literature explores model-based approaches <cit.> and deep learning methods <cit.>.
While model-based methods leverage image priors, challenges persist in obtaining high-fidelity, low-distortion HR-HSI due to the lack of large-scale training datasets. Among deep-learning approaches, CNN-based networks for HR-MSI and LR-HSI tend to be limited and lack interpretability for MHIF tasks and Transformer frameworks <cit.> address the small receptive field of CNN but bring greater computational overhead.


In recent years, implicit representations of 3D scenes have garnered significant attention from researchers. For instance, Neural Radiance Field <cit.>  models 3D static scenes by mapping coordinates to signals through a neural network. Inspired by this, researchers have revisited image representation for 2D tasks. Recent studies <cit.> have achieved arbitrary-scale super-resolution (SR) by replacing commonly used upsampling layers with local implicit image functions. Though these methods demonstrate superior performance in 2D tasks, they still have some drawbacks. Firstly, INR calculates the RGB values of a queried coordinate based on the relative distances to the surrounding four pixels, treating it as a local operation in space that lacks consideration for global information. Additionally, the MLP-ReLU structure used in traditional INR inherent high-frequency information bias <cit.> which is challenging to be eliminated during training.






To address these issues, we propose implicit fusion functions tailored for the MHIF task as a novel fusion paradigm. We first employ encoders to extract prior information from LR-HSI and HR-MSI, which is then fed into the implicit fusion functions in the form of latent codes. Unlike traditional INR, we transform latent codes into the Fourier domain and simultaneously perform spatial and frequency fusion in a unified network. This approach not only rectifies the high-frequency insensitivity induced by the MLP but also effectively extends the receptive field, encompassing a more comprehensive scope of global information. To integrate spatial and frequency domain representations efficiently, we design a decoder with time-frequency tightness, mapping features on both domains to pixel space. The contributions of this work are three folds:

    
  * We define a novel fusion framework based on INR, which innovatively extracts information from the spatial and  Fourier domains, effectively enhances the representation ability of high-frequency information, and expands the receptive field.
    
    
    
    
  * We propose a new decoder employing a Gabor wavelet activation function to enhance the interaction of INR features. Furthermore, we theoretically prove that the complex Gabor wavelet activation possesses a time-frequency tightness property, which facilitates the decoder in learning the optimal bandwidths.
    
    
    
    
  * The proposed network reaches state-of-the-art (SOTA) performance on the MHIF task across two widely used hyperspectral datasets at various fusion ratios. Fig. <ref> provides a fair comparison with other SOTA methods.




§ RELATED WORKS








 §.§ Implicit Neural Representation (INR)

Unlike traditional discrete representations, neural implicit representation (INR) provides a more elegant and continuous parameterized approach. Initially applied in 3D modeling tasks, NeRF <cit.> revolutionized 3D computer vision by representing intricate three-dimensional scenes with just 2D pose images.

This line of work extends to the 2D imaging domain, where INR performs a weighted average on adjacent sub-codes to ensure output value continuity. LIIF <cit.> recently introduces a local implicit image function for SR, leveraging MLP to sample pixel signals across the spatial domain. Several improvements focus on decoding networks; for example, UltraSR <cit.> incorporates residual networks, merging spatial coordinates and depth encoding. DIINN <cit.> utilizes a dual-interactive implicit neural network to decouple content and position features, improving decoding capabilities. JIIF <cit.> proposes joint implicit image functions for multimodal learning, extracting priors from guided images.
Regarding activation functions in the MLP,  SIREN <cit.> recommends utilizing periodic activation functions for continuous INR to fit complex signals.
On the other hand, WIRE <cit.> further employs continuous complex Gabor wavelet activation functions to activate non-linearity, focusing more on spatial frequencies. 

However, there is limited research dedicated to designing INR architectures specifically for the MHIF task. The unique characteristics of hyperspectral images pose challenges for INR networks, in their insensitivity to high-frequency information.




 §.§ Latent Enhancement by Fourier Transform

Fourier transform is a commonly used time-frequency analysis technique in signal processing, which converts signals from the time domain to the frequency domain. The Fourier domain has global statistical properties, and in recent years, many researchers have focused on processing frequency domain information in Fourier space.  Many works use the Fourier transform to enhance the representation ability of neural networks. For example, FDA <cit.> proposes exchanging amplitude and phase components in Fourier space between images to enhance and adjust frequency information. FFC <cit.> introduces a novel convolution module that internally fuses cross-scale information to capture global features in Fourier space.
Similarly, GFNet <cit.> uses 2D discrete Fourier transform and inverse transform to extract features, implements learnable global filtering, and replaces the self-attention layer in Transformer.
UHDFour <cit.> embeds Fourier transform into the image enhancement network to model global information. Together, these studies demonstrate the utility of frequency domain information in improving performance on visual tasks. We exploit the architecture of FeINFN to transform latent codes into the frequency domain, implicitly integrating representations of amplitude and phase components, and enhancing high-frequency injection.



 §.§ Motivation




<cit.> finds that most neural networks exhibit a phenomenon of spectral bias through Fourier analysis. This includes neural networks such as MLP, which tend to learn low-frequency information during the early stages of training and are insensitive to high-frequency information. Moreover, we found this issue occurs in the MHIF task according to an experimental analysis as shown in Fig. <ref>, where HR-HSI and LR-HSI were concatenated with HR-MSI and fed into a trained encoder to obtain latent codes. These codes were transformed into the frequency domain to visualize the amplitude and phase. 

It can be observed that the amplitudes from HR-HSI and LR-HSI are very similar, while the phases differ significantly. The phase of HR-HSI should naturally contain more texture than LR-HSI, a hypothesis validated by the visualized phase maps. Based on this finding, we transformed the latent codes into the Fourier domain to separately process amplitude and phase, to enhance the global learning of high-frequency information in the images. 



§ METHODOLOGY

In this section, we first present the preliminary of INR  and then provide the proposed framework tailored for MHIF task. Subsequently, we elaborate on the implementation details of the composited modules of the proposed FeINFN.



 §.§ Preliminary: Implicit Neural Representation


Neural Radiance Fields <cit.> is represented by integral construction scenes. The value of a pixel in a certain viewing angle image is regarded as the integral of the characteristics of the sampling point from the proximal end to the far end of the ray. During actual training, the integral needs to be discretized. Extended to 2D image representation <cit.>, it is sampled pixel by pixel from the vicinity of the query target.
Taking the low-resolution (LR) image 𝐈∈ℝ^h × w × 3 upsampling to the high-resolution (HR) image 𝐈̂∈ℝ^H × W × 3 as an example, the process of generating the RGB values of the target coordinates 𝐱_q∈ℝ^2 can be regarded as interpolation form, expressed as:


    𝐈̂(𝐱_q)=∑_i∈𝒩_qw_q,i𝐯_q,i,

where 𝐯_q,i∈ℝ^4 × 4 × 3 is the interpolation pixel of i interpolated by q's surrounding pixels 𝒩_q∈ℝ^4 and w_q,i∈ℝ signifies the interpolation weight. In the implicit representation of local image features, the weights w_q, i=S_i / S, where S_i represents the area formed by q and i in the diagonal region and S denotes the total area enclosed by the set 𝒩_q.

The interpolation value 𝐯_q,i is effectively generated by a basis function:

    𝐯_q,i = ϕ_θ(𝐳_i,𝐱_q-𝐱_i),

where ϕ_θ is typically an MLP, 𝐳_i is the latent code generated by an encoder for the coordinates 𝐱_i, and 𝐱_q-𝐱_i represents the relative coordinates. 
From the above equations, it can be inferred that the interpolation features can be represented by a set of local feature vectors in the LR domain.
Typically, interpolation-based methods <cit.> achieve upsampling by querying 𝐱_q-𝐱_i in the arbitrary SR task. See more details in <cit.>.



























 §.§ Overview of the FeINFN Framework

In this work, we propose the FeINFN, which adopts a novel framework for simultaneously performing neural implicit representation in both the spatial and frequency domains to execute the MHIF task. Fig. <ref> provides an overview of the proposed framework, designed to fuse LR-HSI 𝐈^LR∈ℝ^h × w × S and HR-MSI 𝐈^HR∈ℝ^H × W × s to generate HR-HSI images 𝐈∈ℝ^H × W × S based on a given upsampling scale r. 

Initially, the LR-HSI is fed into encoder E_χ to extract spectral features  𝐙_spe∈ℝ^h × w × C. Simultaneously, the concatenated bicubic interpolation LR-HSI 𝐈_up^LR∈ℝ^H × W × S and 𝐈^HR, are fed into encoder E_ψ to extract spatial features 𝐙_spa∈ℝ^H × W × C. Additionally, the pixel's central position is represented as the coordinate point. The coordinate map is normalized into a two-dimensional grid [-1,1]× [-1, 1], obtaining a HR normalized 2D coordinate map 𝐗^HR∈ℝ^H × W × 2. The extracted 𝐙_spe and 𝐙_spa, along with the 2D coordinates of 𝐈^HR, are forwarded to Spatial-Frequency Implicit Fusion Function (Spa-Fre IFF), outputting spatial domain features ℰ_s∈ℝ^H × W × S and frequency domain features ℰ_f∈ℝ^H × W × S. The ℰ_s and ℰ_f serve as inputs to a pixel space mapping decoder which generates the residual image 𝐈_r^HR∈ℝ^H × W × S. Finally, the residual image 𝐈_r^HR is combined with the bicubicly upsampled image 𝐈_up^LR via element-wise addition, yielding the ultimate fusion image 𝐈.




 §.§ INR Encoder Networks
  
Analogous to local implicit representation functions <cit.>, the initial step involves extracting latent code representations.
For the MHIF task, we address the challenges of both upsampling and fusion simultaneously, employing implicit neural representations as the solution.
The INR encoders try to extract spatial and spectral latent codes 𝐙_spa∈ℝ^H × W × C, 𝐙_spe∈ℝ^h × w × C:
one is extracted from 𝐈^LR, serving as the carrier for spectral information; the other is encoded from the concatenation of 𝐈_up^LR and 𝐈^HR, aiding in spatial information during the fusion process. This process can be denoted as:

    𝐙_spe=E_χ(𝐈^LR),
    𝐙_spa=E_ψ(Cat(𝐈_up^LR,𝐈^HR)),

where E_χ is the spectral encoder parameterized by χ, E_ψ is the spatial encoder parameterized by ψ, and Cat(𝐈_up^LR, 𝐈^HR) denotes the concatenation along the channel dimension. In practice, we utilize  EDSR <cit.> as INR encoder networks.



 §.§ Spatial-Frequency Implicit Fusion Function

To address the mentioned issues <ref>, we propose Spatial-Frequency Implicit Fusion Function, dubbed Spa-Fre IFF which is a dual-branch fusion function and utilized for computing the fusion feature of 𝐙_spe and 𝐙_spa in the spatial and frequency domains, respectively. Given a queried HR coordinate 𝐱_q∈𝐗^HR of a pixel unit q, Spa-Fre IFF estimates spatial feature vector ε_s∈ℝ^1×1× S (ε_s∈ℰ_s) and frequency feature vector ε_f∈ℝ^1×1× S (ε_f∈ℰ_f)as follows:

    ε_s,ε_f=Spa-Fre IFF(𝐳_spe,𝐳_spa,δ𝐱),

where 𝐳_spe∈ℝ^1×1× C represents the spectral latent code vector corresponding to 𝐱_q, and 𝐳_spa∈ℝ^4×4× C 
 is the spatial latent code vector. δ𝐱 denotes the set of local relative coordinates, expressed by the following formula:

    δ𝐱={𝐱_q-𝐱_q,i}_i∈𝒩_q,

where 𝐱_q,i refers to the coordinates most proximate to the query coordinate 𝐱_q, representing the four corner pixels closest to q in the HR space.

Spatial Implicit Fusion Function: The Spatial Implicit Fusion Function aims to leverage the powerful representation capabilities of INR to achieve implicit fusion in the spatial domain, as shown in Fig. <ref> (see branch “Spatial Domain”). Specifically, we employ high-pass operators ℋ to filter the spectral latent codes, 

as a complement to the high-frequency information on the spectrum:

    𝐳_hp=ℋ(𝐳_spe) ,

where 𝐳_hp∈ℝ^1×1× C represents the high-frequency latent code of 𝐈^LR.
Also, we suggest frequency encoding for relative positional coordinates as follows:

    γ(δ𝐱)=   [sin(2^0δ𝐱),cos(2^0δ𝐱),⋯,
        
       sin(2^L-1δ𝐱),cos(2^L-1δ𝐱)] ,


where L is a hyperparameter, in practice, we set L to 10.

Additionally, leveraging the graph attention mechanism <cit.>, we parameterize the solution for interpolation weights 𝐰_q, i∈ℝ^1 × S, and the implicit fusion function simultaneously outputs fusion interpolation values 𝐯_q, i∈ℝ^4 × 4 × S and interpolation weights 𝐰_q, i. The implicit fusion function is specifically expressed as:

    𝐰_q,i,𝐯_q,i=ϕ_θ(𝐳_spe,𝐳_spa,𝐳_hp,γ(δ𝐱)) ,

where ϕ_θ is an MLP parameterized by θ.
The spatial implicit fusion interpolation, as shown in Eq. (<ref>), yields the fused spatial feature ε_s∈ℝ^1×1× S and can be described as follows:

    ε_s=∑_i∈𝒩_q𝐰_q,i* 𝐯_q,i .







Frequency Implicit Fusion Function: From Fig. <ref>, 

we observed characteristics in the frequency features between LR-HSI and HR-HSI. Hence, we design a frequency implicit fusion function to express global features continuously in the Fourier domain. Notably, directly applying static kernel convolution in the frequency domain would only enhance a specific frequency range, which is inappropriate for the fusion task. However, by learning feature content to generate weights, INR can be seen as a dynamic interpolation method in continuous space, adaptively enhancing information in the frequency domain without overly altering the frequency distribution. Therefore, introducing INR into the Fourier domain is reasonable. Since amplitude and phase exhibit different forms, as shown in Fig. <ref>, we handle them separately.

With the considerations mentioned above, as illustrated in Fig. <ref> (see branch “Fourier Domain”), we initially employ FFT to transform latent codes 𝐳_spe and 𝐳_spa from the spatial domain to the frequency domain, obtaining 𝐟_spe∈ℝ^1×1× C and 𝐟_spa∈ℝ^4×4× C.



After the transformation, we further obtain amplitude components 𝒜(𝐟_spe) and 𝒜(𝐟_spa),  as well as phase components 𝒫(𝐟_spe) and 𝒫(𝐟_spa).

For the amplitude, as shown in Fig. <ref>, the amplitude distribution of LR-HSI and HR-HSI are very similar, and the non-point-wise convolution (e.g. Conv 3× 3) causes an issue of spectrum leakage, confusing channel information. 
In contrast, point-wise convolution does not span multiple locations in the frequency domain and has no overlap allowing it to capture information across channels effectively.
Thus the fusion function for amplitude components is more suitable when applying point-wise convolution:

    𝐰^𝒜_q,i,𝐯^𝒜_q,i=ϕ_α^𝒜(𝒜(𝐟_spe),𝒜(𝐟_spa),δ𝐱 ) ,

where 𝐰^𝒜_q, i∈ℝ^1 × S and 𝐯^𝒜_q, i∈ℝ^4×4× S   are the weights and interpolated values for the corresponding amplitude component, and ϕ_α^𝒜 is a simple network composed of two layers of point convolutions parameterized by α.

Similar to operations in the spatial domain,

implicit fusion interpolation is performed after obtaining interpolated values 𝐯^𝒜_q, i and the normalized weights 𝐰^𝒜_q,i:

    𝒜^'_f=∑_i∈𝒩_q𝐰^𝒜_q,i
    *𝐯^𝒜_q,i ,

where 𝒜^'_f∈ℝ^1×1× S is the integrated amplitude component.

For the phase, which encapsulates information such as texture details, LR-HSI and HR-HSI often have different phase information. It is known that point convolutions fail to capture sufficient spatial representations. Therefore, we use a 3× 3 convolution to learn phase information. Additionally, small changes in the frequency domain may result in significant variations in the spatial domain.

We still consider using the form of INR interpolation for phase learning. The handling of the phase components 𝒫(𝐟_spe) and 𝒫(𝐟_spa) are formally similar to Eqs. (<ref>) and (<ref>):


    𝐰^𝒫_q,i,𝐯^𝒫_q,i   =ϕ_β^𝒫(𝒫(𝐟_spe),𝒫(𝐟_spa),δ(𝐱)) ,
    𝒫^'_f   =∑_i∈𝒩_q𝐰^𝒫_q,i* 𝐯^𝒫_q,i.


The simple network ϕ_β^𝒫 consists of two layers of 3× 3 convolutions parameterized by β. 𝒫^'_f∈ℝ^1× 1 × S represents the integrated phase component.

Finally, IFFT is applied to map the frequency features 𝒜^'_f and 𝒫^'_f back to the image space, obtaining the frequency domain feature ε_f∈ℰ_f. Since in frequency space, one frequency point may correspond to multiple pixels at different positions in the spatial domain, the receptive field of INR in the frequency domain is enlarged in the spatial domain.




 



 §.§ Spatial-Frequency Interactive Decoder



After obtaining the spatial feature map and frequency domain feature map, it is essential to consider how to integrate them seamlessly. Firstly, our decoder needs to have dual input and interactive capabilities. Secondly, it is necessary to focus on representing images in the spatial-frequency domain. With this in mind, we introduce the complex Gabor wavelet activation function with good time-frequency tightness and propose the Spatial-Frequency Interactive Decoder (SFID). Specifically, SFID consists of three layers, taking spatial and frequency domain features as inputs. The outputs 𝐈^HR_r and 𝐈^HR_up contribute to the final fused image 𝐈. The decoding process is illustrated in Fig. <ref>.









The complex Gabor wavelet function is defined as:

    𝒢(𝐱)=e^jω_0𝐱e^-|υ_0𝐱|^2,

where ω_0 is the center frequency in the frequency domain, υ_0 is a constant that is considered as the standard deviation of the Gaussian function, and 𝐱 is a vector in the time (or spatial) domain. In what follows, we provide a theorem below that this Gabor wavelet activation has time-frequency tightness <cit.>, which is helpful for the decoder's information interaction.

Theorem 1.
The complex Gabor wavelet activation in Eq. (<ref>) has the time-frequency tightness property. Moreover, from the perspective of signal spectrum analysis, this activation helps the decoder learn the optimal bandwidths.

Proof: The detailed proof can be found in the Supplementary.



















§ EXPERIMENTS

Datasets:
To evaluate the efficacy of our model, we conducted experiments using the CAVE and Harvard datasets. The CAVE dataset comprises 32 Hyperspectral Images (HSIs) with 31 spectral bands spanning from 400 nm to 700 nm at 10 nm intervals. We randomly selected 20 images for training and used the remaining 11 for testing. The Harvard dataset consists of 77 HSIs depicting indoor and outdoor scenes, covering the spectral range from 420 nm to 720 nm. We standardized the data by cropping the upper left sections of 20 Harvard images, with 10 for training and the rest for testing. Details can be found in the Supplementary.





Implementation details:
We implement the proposed methods FeINFN with Pytorch <cit.> on a workstation with an Intel I9 CPU and two 3090 GPUs. The optimizer is chosen as AdamW <cit.> and we use a Cosine anneal learning rate scheduler. The base channel number of the encoder is 128, that of the proposed implicit fusion function is 32 and in the decoder, the channel number is 31.

Benchmark:
To evaluate FeINFN's performance, we compare it with  MHIF methods on the CAVE and Harvard datasets. The bicubic-interpolated result of the upsampled LR-HSI in Tab. <ref> serves as our baseline. Various model-based techniques, including the CSTF-FUS  <cit.>, LTTR <cit.>, LTMR <cit.>, and IR-TenSR <cit.> approaches, are considered. Additionally, we compare our approach with various deep learning methods, such as SSRNet <cit.>, ResTFNet <cit.>, HSRNet <cit.>, MoGDCN <cit.>, Fusformer <cit.>, and DHIF <cit.>, PSRT <cit.>, 3DT-Net <cit.>, DSPNet <cit.>, BDT <cit.>. 

We compare our method with other methods using different image quality metrics to validate the image fusion capability of our model, including SAM <cit.>, ERGAS <cit.>, PSNR <cit.>, and SSIM <cit.>.

Results on CAVE Dataset:
In this section, we evaluate the effectiveness of FeINFN on the CAVE dataset and compare it with five traditional methods and some state-of-the-art deep learning-based approaches. As shown in Tab. <ref> on the left, our method achieves optimal performance in the tasks of × 4 in all metrics. In the × 4 experiment, compared to currently leading methods such as DSPNet <cit.>, 3DT-Net <cit.>, and BDT <cit.>, our approach demonstrates improvements in PSNR by 1.29dB/1.09dB/0.17dB, respectively. The × 8 experiment is detailed in the Supplementary. 

To illustrate the advantages of our method, we provide visual comparisons in Fig. <ref>, including close-ups and error maps to highlight specific details. Our fusion results closely match the ground truth, achieving the best quality. In comparing error maps, the darker colors indicate closer proximity to the original image. In contrast to other excellent methods, the error maps of FeINFN distinctly exhibit superior restoration effects on details.

Results on Harvard Dataset:
In Tab. <ref>, the right columns present the comparison results of our FeINFN with other methods on the Harvard dataset at scale factors 4. Our method performs exceptionally well, with only SAM being slightly surpassed by 3DT-Net <cit.> and BDT <cit.>. FeINFN exhibits significant gains in PSNR/ERGAS/SSIM metrics compared to the current state-of-the-art <cit.>, with improvements of 0.14dB/0.16/0.001, respectively. The results with a scale factor of 8 can be found in the Supplementary. As depicted in Fig. <ref>, our model outperforms others, highlighting the crucial role of FeINFN's continuous representation capability in high-scale factor scenarios. To better visualize the performance gap, Fig. <ref> illustrates the fused images and error maps, confirming that our FeINFN maintains high fidelity in recovering the texture details of the images.










				

				

				


				


				


				


				

    


				


				


				


				

				

				

				





















 





 §.§ Ablation Studies

Upsampling methods:

 Implicit image representation can be seen as an advanced interpolation algorithm, offering additional spatial information and parameterized weight generation. In this section, we compare INR with other upsampling methods. We replace INR with pixel-shuffle <cit.> and traditional CNN interpolation methods, presenting a comparative analysis. As seen in Tab. <ref>, our approach outperforms other methods in MHIF tasks.




Spatial domain and Fourier domain:
To assess the dual-domain model's efficacy, we performed model reduction, preserving spatial and Fourier domains independently. As shown in Tab. <ref>, FeINFN excels by using both spatial and Fourier domains concurrently, underscoring the positive impact of Fourier domain integration on overall network performance.

Spectral deviation occurs during training, where the network tends to prioritize low-frequency information, capturing high-frequency details only in later stages. To validate our resolution of this issue, we remove the “Fourier Domain" from Spa-Fre IFF, or retain it, and the corresponding training data is illustrated in Fig. <ref>. Our FeINFN, which incorporates Fourier domain fusion, leads to faster PSNR convergence and overall higher efficiency. The visual comparison of high-frequency details in “chart and stuffed toy" from the cave dataset at 80k iterations further supports the significant improvement achieved with our results. 


Decoder with Different Nonlinear:
In this section, we evaluate the impact of different activation functions in SFID, aiming to match SFIFF. Our dual-input decoder incorporates a complex Gabor wavelet activation function to facilitate the fusion of spatial and frequency domain features.

Through experiments, we replaced the Gabor wavelet activation with other activations, presenting the results in Tab. <ref>. The findings distinctly demonstrate the enhanced fusion quality achieved with the complex Gabor wavelet activation. This emphasizes the critical role of wavelet activation in promoting robust and reliable learning in SFID.


§ CONCLUSION


Inspired by the distinct behaviors of LR-HSI and HR-HSI in the Fourier domain, we introduce a novel Fourier-enhanced Implicit Neural Fusion Network (FeINFN) based on INR. Through Fourier transformation, latent features are converted into the frequency domain, allowing the modeling of frequency components to enrich high-frequency information in images. 
Additionally, we propose a spatial-frequency decoding module, achieving a unified representation of both spatial and frequency domains using a time-frequency-tight activation function.
Thanks to the unique design of our network, it outperforms state-of-the-art methods in MHIF with appealing efficiency. 
We desire that our work will inspire future research on frequency fusion-based MHIF methods.






named


