Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and Restoration for CubeSat
Chih-Chung Hsu Chih-Yu Jian Eng-Shen Tu Chia-Ming Lee Guan-Lin Chen

























  


 





⌈⌉
⌊⌋



 









@@
	@math@cr
	@tag
	10 @
	12 @
	@@8 @@
	@th##
_@











myi

		
			  
	
























FactFact
LemmaLemma
PropProposition
TheoremTheorem
DefDefinition
CorollaryCorollary
PropertyProperty
ObservationObservation

ExaExample
RemarkRemark


Ventry[1]

			
			
			
			
	









































































op-tical net-works semi-conduc-tor



















	
	Real-Time Compressed Sensing for Joint Hyperspectral Image Transmission and Restoration for CubeSat

	
	This study was supported in part by the National Science and Technology Council (NSTC), Taiwan, under grants MOST 112-2221-E-006 -157 -MY3 and 111-2221-E-006 -210; partly by the Higher Education Sprout Project of Ministry of Education (MOE) to the Headquarters of University Advancement at National Cheng Kung University (NCKU). We thank to National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.

(Corresponding author: Chih-Chung Hsu.)


		C.-C. Hsu, C.-Y. Jian, C.-M. Lee and G.-L. Chen are with Institute of Data Science and Department of Statistics, Miin Wu School of Computing, National Cheng Kung University, Tainan, Taiwan (R.O.C.), (e-mail:cchsu@gs.ncku.edu.tw, zuw408421476@gmail.com, alright1117@gmail.com)
		E.-S. Tu is with Department of Computer Science, National Cheng Kung University, Tainan, Taiwan (R.O.C.)

    Chih-Chung Hsu, Senior Member, IEEE, Chih-Yu Jian, Eng-Shen Tu, Chia-Ming Lee and Guan-Lin Chen  <https://rcts01.github.io/>
    
================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




        

            This paper addresses the challenges associated with hyperspectral image (HSI) reconstruction from miniaturized satellites, which often suffer from stripe effects and are computationally resource-limited. We propose a Real-Time Compressed Sensing (RTCS) network designed to be lightweight and require only relatively few training samples for efficient and robust HSI reconstruction in the presence of the stripe effect and under noisy transmission conditions. 
            
            The RTCS network features a simplified architecture that reduces the required training samples and allows for easy implementation on integer-8-based encoders, facilitating rapid compressed sensing for stripe-like HSI, which exactly matches the moderate design of miniaturized satellites on push broom scanning mechanism. This contrasts optimization-based models that demand high-precision floating-point operations, making them difficult to deploy on edge devices. Our encoder employs an integer-8-compatible linear projection for stripe-like HSI data transmission, ensuring real-time compressed sensing. Furthermore, based on the novel two-streamed architecture, an efficient HSI restoration decoder is proposed for the receiver side, allowing for edge-device reconstruction without needing a sophisticated central server. This is particularly crucial as an increasing number of miniaturized satellites necessitates significant computing resources on the ground station. 
            
            Extensive experiments validate the superior performance of our approach, offering new and vital capabilities for existing miniaturized satellite systems. 
            
		
		Index Terms—
		deep learning, 
		hyperspectral image, 
		compressed sensing,
		hyperspectral restoration,
		real-time applications.
		
	
	
	


	


	
	





§ INTRODUCTION

    
Hyperspectral remote sensing benefits tasks including environmental monitoring, matter recognition, urban planning, agriculture, and surveillance. Deep learning (DL), growing in popularity for vision tasks, has significantly impacted hyperspectral remote sensing applications <cit.>, demonstrating superior performance over conventional optimization-based approaches <cit.>.

Hyperspectral images (HSI) contain hundreds of spectral bands, both visible and invisible, resulting in a substantial data volume. Efficient remote sensing is essential for transmitting HSI to ground receivers for applications in matter recognition, urban planning, and agriculture. The increase in Low Earth Orbit satellites (LEOS) based on CubeSat/miniaturized technology has led to bandwidth challenges due to substantial data transmission. Hyperspectral compressed sensing (HCS) is thus widely used for concurrent sensing and compression of HSI <cit.>.


Hyperspectral sensors capture data in stripe-like sequences, forming HSI in the along-track direction, a method known as pushbroom sensing. However, satellite movement, orography, or IFOV (Instantaneous Field of View) issues mean that spatial correlation between contiguous slices may not always be guaranteed. Thus, we must perform geometric correction and geographic data projection to achieve spatially correlated images. Given its high computational load, this pre-processing is often deferred to post-transmission at the ground station. Moreover, limited memory in miniaturized satellites restricts the stacking of multiple spectral slices. Stripe-like encoding, as proposed in DCSN <cit.>, offers a solution by encoding fewer slices, emphasizing the need for efficient and precise stripe-like hyperspectral data transmission from miniaturized satellites to ground receivers.

Conventional data compression strategies like JPEG, JPEG2000, and video codecs could theoretically reduce HSI data volume. Several approaches have been proposed, utilizing principal component analysis (PCA) and codecs such as JPEG2000 <cit.> and X264 <cit.>. However, given the prevalent use of pushbroom sensing for acquiring HSI (stripe-like data) and the limited storage of miniaturized satellites, traditional image/video compression techniques may lack the necessary computational resources <cit.>, rendering these methods incompatible for real-time compressed sensing in such satellites.

The power and computational limitations in CubeSats necessitate the adoption of lightweight and efficient HCS. HCS can perform concurrent compression and sensing, significantly easing computational and storage demands in miniaturized satellites. Therefore, an ultra-lightweight HCS is crucial. Spectral compressive acquisition (SpeCA) <cit.> leverages spatial and spectral redundancies to create random projection matrices for low-dimensional data extraction. Novel HCS methods utilizing L_1-norm and total-variation (TV) regularization have been introduced <cit.> to exploit global spatial-spectral correlations in HSI data while maintaining local smoothness. A spatial/spectral compressed encoder (SPACE), based on low-rank representation <cit.>, transforms HCS into an image fusion process by separately compressing spatial and spectral data. The recently improved SPACE variant, All-Addition Hyperspectral Compressed Sensing (AAHCS), enhances both spectral and spatial qualities of reconstructed HSI data <cit.>. However, these optimization-based HCS methods often require lengthy computational times due to the necessity of eigendecomposition. Although AAHCS demonstrates fast sensing and high performance through all-addition operations in the encoder <cit.>, real-time compressed sensing is hampered by the required eigendecomposition, which is computationally intensive (𝒪(n^3)) and thus challenging for on-board processing in miniaturized satellites. Moreover, the stripe-like HSI acquired through pushbroom sensing in moderate satellites does not align well with these optimization-based HCS designs, potentially suppressing performance.

Deep learning offers an effective and efficient solution for hyperspectral compressed sensing (HCS) challenges. The recently introduced Deep Compressed Sensing Network (DCSN) addresses stripe-like HSI data processing challenges in miniaturized satellites <cit.>. DCSN features a non-square kernel within a three-layer convolutional neural network (CNN), creating a lightweight encoder that swiftly compresses HSI data before transmission to the ground station. For data recovery, DCSN employs a complex decoder leveraging Multi-scale Feature Fusion Block (MFB) and Aggregation (MFA).

Despite DCSN's state-of-the-art performance at lower sampling rates (approximately 1%), achieving real-time HCS is hindered by delays from multi-layer CNNs with non-linear activation functions. Conventional activation functions, such as ReLU, demand additional power and computational resources <cit.>. The hardware implementation of these functions requires multiple units to approximate non-linear models, leading to increased computation time <cit.>. Consequently, the LeakyReLU <cit.> activation function used in DCSN's encoder may not be practical for resource-constrained devices, like edge devices <cit.>.
 
Minimizing the computational complexity of HSI data decoding is crucial. The surge in edge computing, which involves processing on mobile or low-power embedded devices, is increasingly popular due to its ability to reduce central server load <cit.>. Quick reconstruction of compressed hyperspectral data on these devices, including geometric correction and stitching, alleviates decompression burdens on central servers. However, deploying sophisticated decoders like those in DCSN <cit.> on edge devices is challenging. Additionally, the iterative decoding approach of AAHCS <cit.> precludes real-time processing on such devices. As the number of LEOS is expected to rise, simplifying the decoder's computational demands becomes essential for real-time applications.

A critical yet often overlooked issue in hyperspectral compressed sensing (HCS) is the striping effect caused by anomalies in hyperspectral sensor's electrical units <cit.>. While robust HCS methods like DCSN <cit.>, AAHCS <cit.>, SPACE <cit.>, and SpeCA <cit.> typically focus on channel noise during transmission, they seldom address the striping effect. This phenomenon, often handled as a data imputation challenge and addressed through HSI inpainting <cit.>, is especially prevalent in devices like NASA's EO-1 satellite's Hyperion imager <cit.>. Addressing this striping effect in HCS is essential for enhancing the robustness of sensing techniques in miniaturized satellites, aiming to minimize stripe artifacts in the reconstructed HSI.



This manuscript presents the Real-time Compressed Sensing (RTCS) network, a system innovatively designed for the efficient transmission and restoration of hyperspectral data. It addresses challenges such as rapid sensing, stripe effect mitigation, and noise-resistant transmission. Figure <ref> demonstrates how RTCS adeptly balances efficiency and effectiveness in real-time applications. In our evaluations, conducted on the Jetson TX2 platform, RTCS and DCSN <cit.> displayed notable performance, while other HCS methods were tested on a personal computer due to their incompatibility with Arm-based architectures (refer to Section <ref>). In contrast to DCSN's <cit.> reliance on multi-layer nonlinear activations, which can impede real-time processing, RTCS employs a straightforward linear projection. This approach effectively captures essential spatial and spectral features, resonating with the success of linear models in optimization-based HCS <cit.>. Furthermore, the encoder in RTCS is tailored for hardware efficiency and compatible with quantization, unlike the high-precision requirements of traditional optimization-based methods.

With the burgeoning number of LEOS, real-time decoding on edge devices is becoming increasingly crucial. Deeper networks typically result in slower inference due to inter-layer dependencies. To address this, our proposed decoder features a reduced depth to facilitate faster inference. It incorporates a novel two-streamed network with a Cross-Scale Feature (CSF) module, which efficiently captures a range of features across different scales. We employ grouped convolution to significantly cut down on the number of parameters and computational complexity, taking advantage of the redundancy in HSI data spectra. This innovation enables the decoder to support real-time HSI reconstruction on edge devices. Additionally, we introduce a Spectral Angle Mapper-aware (SAM) Loss, aiming to enhance spectral quality by guiding spectral signature reconstruction. The RTCS, with its task-specific training approach, demonstrates exceptional resistance to stripe effects commonly encountered in miniaturized satellites. Moreover, the unique two-streamed network design, combined with RTCS's streamlined architecture and reduced parameter count, significantly boosting its adaptability and training efficiency for various hyperspectral sensors.

The primary contributions of this paper are as follows:



  * Simple Linear Projection and Edge Device Compatibility in encoder:
RTCS introduces a unique linear projection in its encoder, precisely aligning with the principles of compressed sensing. This projection not only adheres to the integer-8 standard, making it highly suitable for edge devices, but also offers significant power savings and computational reduction, deviating from traditional optimization approaches.


  * Innovative Two-Streamed Network in decoder:
The novel two-streamed network architecture in RTCS employs a lightweight design that effectively balances performance and robustness. This innovative approach enhances the system's capabilities without imposing additional computational complexity, exemplifying an advanced integration of efficiency and functionality.



  * Task-Specific Training for Enhanced Capabilities:
The RTCS leverages a task-specific training policy, enabling it to excel not only in compressed sensing for efficient transmission but also in reducing stripe effects in decoded hyperspectral images. This marks the first instance of seamlessly integrating compressed sensing and tensor inpainting within a single model, achieving unparalleled performance across extensive datasets.




The remainder of this paper is organized as follows.
Section <ref> introduces the proposed RTCS architecture design.
Section <ref> experimentally demonstrates the superiority of RTCS over benchmark methods. Finally, Section <ref> presents the conclusions drawn from this study.
    
    
        
    
	
	
	


§ PROPOSED METHOD
 
	
    



As illustrated in Figure <ref>, we present a novel transmission framework designed to minimize the computational complexity inherent in the encoding and decoding processes. This enables the swift decoding of hyperspectral data on edge devices, thereby alleviating the computational burden on central servers. Such a framework gains prominence as the constellation of Low Earth Orbit (LEO) satellites, including CubeSats, expands. It allows central servers to allocate their computational power more efficiently to demanding tasks, like geometric correction or hyperspectral data integration, rather than spending it on decoding. Thus, the ability to perform encoding and decoding in real-time is crucial.

The Real-Time Compressed Sensing (RTCS) method presented in this study is specifically designed for use in compact satellite systems. It introduces a stripe-based compressed sensing technique that is compatible with the sensing capabilities of modern miniaturized satellites and is optimized for low memory usage. This approach leverages simple matrix operations to facilitate real-time onboard satellite computations. The RTCS's encoder operates through a linear transform-based method that is both hardware-friendly and conducive to integer operations, contrasting with traditional Hyperspectral Compressed Sensing (HCS) that relies on high-precision floating-point operations. As a result, our encoder is well-suited for edge devices with limited processing power. The RTCS's support for stripe-like HCS significantly reduces memory demands on miniaturized satellites, requiring only four hyperspectral stripes for effective compression and immediate transmission. Even on satellites with stringent resource limitations, RTCS maintains its capability to encode hyperspectral data efficiently.



Ground station decoding speed is a critical factor. With an increasing fleet of Low Earth Orbit Satellites (LEOS), the central server may be inundated with compressed hyperspectral data awaiting decoding and integration. The RTCS method we propose circumvents the need for server-centric decoding, instead facilitating the reconstruction of hyperspectral data directly on edge devices within ground stations. This expedites data reconstruction and conserves computational resources for more critical HSI tasks. For this purpose, we introduce a Cross-Scale Feature (CSF) network decoder that employs a dual-stream, shallow architecture, effectively extracting multi-scale features while maintaining computational efficiency. This innovation significantly reduces the workload on central servers.

Moreover, we propose a Spectral Angle Mapper-aware (SAM) Loss to guide the network in preserving spectral signatures, thus enhancing spectral quality. A task-specific training policy is subsequently introduced to improve the robustness of our RTCS, allowing for real-time compressed sensing and reduced stripe effects during the decoding phase. This feature renders the proposed RTCS more practical. Due to the relatively few parameters of the proposed RTCS, relatively few training samples are required to match the promising performance, enhancing feasibility when re-training the RTCS for further applications.

In the ensuing sections, we thoroughly discuss the effectiveness and efficiency of the proposed encoder and decoder in our RTCS.



 §.§ Real-Time Compressed Sensing


The proposed encoder, though simple in design, proves effective by utilizing a single matrix multiplication, similar to the traditional optimization-based HCS that relies on random projection. This approach draws inspiration from the linear subspace/basis concept in optimization-based HCS, which is known to preserve most of the HSI's energy. This forms the basis of our hypothesis that a linear operation in the decoder could suffice for efficient compressed sensing. Specifically, the encoder in our RTCS system consists of a single convolution operation without any activation function, significantly reducing computational complexity and latency. We employ a stripe-like input to align with the pushbroom scanning mechanism typical in moderate LEOS, as indicated in DCSN <cit.>. However, our measurement matrix design in compressed sensing diverges from the encoder in DCSN <cit.>, where a multi-layer approach might lead to considerable encoding delay for hyperspectral stripes and fail to represent a measurement matrix accurately. We contend that single matrix multiplication can act as an efficient encoder, fully meeting the requirements of a measurement matrix in compressed sensing.

Let the original high-resolution (HR) HSI be denoted as _o∈ℝ^B× H× W, where B, H, and W represent the number of bands, height, and width of the HRHSI, respectively. The stripe-like HSI, sensed by the pushbroom mechanism of the LEOS, is defined as = _o,
where R is the sampling matrix used to acquire the stripe-like HSI ∈ℝ^B× h× w, h and w represent the height and width of the stripe-like HSI. Our compressed sensing in the proposed encoder could be done simply by

    Z = Ψ X,

where Ψ is the measurement matrix formed by convolutional and learnable coefficients. Let us explain in detail how the convolution benefits the HCS. Suppose that a low-dimensional representation  of the HSI could be compressively sensed by a single learnable convolution kernel W via 

    Z(m,n) = ∑_j=-k_w/2^k_w/2∑_i=-k_h/2^k_h/2 X(i,j) · W(m-i, n-j),

where k_w and k_h indicate the convolutional kernel's width and height. It is well-known that the convolutional could be expressed as matrix multiplication. Consider a 3× 3 data matrix  and a 2× 2 convolution kernel , the convolution operation (stride s=1, no padding p=0) is equivalent to:


    = ∗ =
    [ x_1 x_2 x_3; x_4 x_5 x_6; x_7 x_8 x_9;     ]∗[ w_1 w_2; w_3 w_4;     ].

This can be reformulated as a matrix multiplication by rearranging  with zero-padding and shifting to form _m and vectorizing  into its column vector form _v:

    = _m_v 
       =
    [ w_1 w_2   0 w_3 w_4   0   0   0   0;   0 w_1 w_2   0 w_3 w_4   0   0   0;   0   0   0 w_1 w_2   0 w_3 w_4   0;   0   0   0   0 w_1 w_2   0 w_3 w_4 ][ x_1; x_2; x_3; x_4; x_5; x_6; x_7; x_8; x_9 ].

In hyperspectral image compression, exploiting spatial and spectral redundancies is essential. Traditional methods typically reshape 3D hyperspectral data ∈ℝ^B× H× W into a 2D matrix _2D∈ℝ^B× (HW) to achieve dimensionality reduction. Linear transformations, for example, efficiently capture spectral information and preserve energy, a principle that can be extended to single convolution layers in deep learning frameworks. This strategy is particularly well-suited for the constraints of miniaturized satellite imaging systems, notably those utilizing pushbroom sensing. It effectively reduces the space complexity that is otherwise inherent in compressing the full tensor.



A concern is that the typical kernel size 3× 3 has a small receptive field in the spatial dimension for HSI. As shown in the convolution example in (3), the compact representation via convolution is determined by only four coefficients W, while traditional optimization-based HCS has a larger number of coefficients for dimensionality reduction. Therefore, a larger kernel size, such as 9× 9, is introduced to capture the long-range dependency in the spatial domain, resulting in better energy preservation. The kernel size is an example of the effective receptive field. To satisfy the requirements of pushbroom sensing in the miniaturized satellites, we use 9× 1 kernel in the encoder to enable the compressed sensing in hyperspectral stripes. Assuming that the input HSI of size B× H× W is projected to its compact representation Z =  X Ψ, ∈ℝ^b× h× w, where b ≪ B, h ≪ H, and w ≪ W, B× k_h× k_w coefficients are required to project the HSI to a single feature map, and thus, b× B× k_h× k_w coefficients are needed to generate the b feature maps for HSI. The complete matrix multiplication diagram of the convolution operation in the proposed encoder is depicted in Figure <ref>, illustrating how convolutional kernels capture the spatial and spectral dependency simultaneously. Specifically, each column vector in the kernel matrix is used to project the row vector of the HSI within a local window determined by kernel size k_h× k_w to the lower dimensional space, where the spectral redundancy is fully explored by B× k_h× k_w coefficients. As the parameter space of the linear convolution is sufficiently large, it should be able to project the HSI to a compact linear subspace.

It is crucial to discuss the computational complexity of the proposed encoder as well. The computational complexity of the proposed encoder, primarily based on matrix multiplication, is 𝒪(b(Bk_w k_h) (hw)) with stride s=1 and padding p=0. In contrast, the major time cost of the most efficient HCS method, AAHCS <cit.>, lies in the basis computation via eigendecomposition, where the complexity can be 𝒪(B^3 + (WH)B(WH)). In AAHCS <cit.>, the vectorized representation _v of HSI  transforms the B× H× W matrix into a B× (WH) matrix and performs eigendecomposition on the square matrix = _v_v^T to obtain the basis Ψ. Although the optimization-based approach <cit.> claims that the basis matrix Ψ can be calculated from pre-collected data, their experiments are evaluated by data-dependent basis (i.e., the basis should be calculated for each HSI individually), resulting in eigendecomposition being required onboard the miniaturized satellite during HCS for optimal performance. In comparison to the computational complexity of basis decomposition in AAHCS <cit.>, 𝒪(B^3 + (WH)B(WH)), the matrix  demands 𝒪((WH)B(WH)). When w ≪ W, h ≪ H, and b≪ B, the computational complexity of the proposed encoder can be significantly reduced. For instance, with a sampling rate of ≈ 1% for our encoder, we set the size of compressed HSI to b=27, h=32, w=1 (cf. Section <ref>) for the HSI sized at 172× 128× 4, resulting in computational complexity of approximately 3.95× 10^6, while the matrix  in AAHSC costs 45.088× 10^6. Furthermore, convolutional operations can be highly parallelized in practice, significantly accelerating the computing process and minimizing onboard computational effort in miniaturized satellites. In summary, the proposed encoder is highly efficient and effective.



 §.§ Hardware-Friendly Decoding and Restoration








As previously stated, decoding compressed HSI  in edge devices can effectively reduce the workload of central servers at ground stations. Toward this goal, we propose a novel network structure for efficient decoding of compressed HSI. While reducing the depth and number of kernels in a neural network is an effective way to accelerate inference, overly simplifying the network can detrimentally affect performance, despite achieving faster inference speeds. For instance, in real-time applications, the MobileNet-based architecture <cit.> is favored due to its computational efficiency. However, there is still a noticeable performance gap between MobileNet and earlier conventional CNN designs like SENet <cit.> and ResNet <cit.>. Specifically, MobileNet's best top-1 accuracy on the ImageNet validation set <cit.> is 75.2%, compared to 78.57% for ResNet-152 <cit.>. In the context of HCS, particularly during the decoding phase, achieving promising performance is as crucial as ensuring speed.


In this section, we highlight the benefits of our proposed network architecture for the decoder, which is fast, hardware-friendly, and effective. This architecture is illustrated in the Figure <ref>. Consider the decoder's input as the compressed HSI ∈ℝ^b× h× w. Our two-stream-based network architecture is designed to learn features with varying receptive fields efficiently. Initially, two stem convolution layers extract low-level features _stem and ζ_stem from Z. In one branch, we implement a modified version of the decoder from DCSN <cit.>, halving its depth and reducing parameters to decrease computational complexity. In the other branch, we propose a Fast Residual-Dense-Block-based Network (FRDB-Net), employing grouped convolution and larger kernels yet maintaining a low parameter size. A Multi-Scale Feature Fusion (MFF) strategy is introduced, combining feature representations from both branches through element-wise addition. Subsequently, using an upsampling block, the low-spatial-resolution (LSR) features are progressively upsampled to high-spatial-resolution (HSR) features. Finally, the HSI is reconstructed from the low-spectral-resolution (LSPR) features using a spectral upsampler via convolution. Detailed descriptions follow in the subsequent paragraphs.


Cross-Scale Feature Network. We first discuss the significant advantages conferred by the two-streamed network architecture in HCS. A direct reduction in network depth can lead to substantial performance loss, as shallower networks often struggle to learn rich feature representations. This is due, in part, to the larger equivalent receptive field of deeper networks compared to shallower ones. Our proposed two-streamed architecture aims to overcome the limitations associated with shallower networks. It comprises two distinct sub-networks utilizing 3× 3 and 9 × 9 convolutional kernels to capture features with different receptive fields. Feature aggregation from these sub-networks is achieved through element-wise addition using a MFF strategy. This approach allows for sharing cross-scale feature information, which is then progressively upsampled in both spatial and spectral dimensions to reconstruct the HSI. 

The top line of Figure <ref> presents the first branch, the Improved Multi-Receptive-field Aggregation Network (IMFA-Net), while the bottom line illustrates the second branch, our newly proposed FRDB-Net with a larger kernel size. In developing the IMFA and FRDB blocks, we opted for a single residual connection to minimize memory usage, addressing the issue of feature map reuse seen in DCSN <cit.>. Our experiments demonstrate that this streamlined approach with just one residual connection is both more effective and efficient.

Specifically, the intermediate feature Z_m could be extracted via

    Z_m = f_u(f_IMFA(f_s_1())) + f_FRDB(f_s_2()),

where f_FRDB and f_IMFA are the backbone networks of the reduced IMFA and the proposed FRDB, f_s_1 and f_s_2 are the stem convolution layers for IMFA and FRDB. In IMFA-Net and the proposed FRDB-Net, we stack n_f IMFA and FRDB blocks, respectively, where n_f=8 in this study. Since the number of the feature maps of f_IMFA is less than that of f_FRDB, an extra convolution layer f_u is used to project the feature map to the desired size. Then, the HSR feature map Z_h could be learned based on the spatial upsampler via

    Z_h = f_up(Bilinear( Z_m)),

where Bilinear stands for the bilinear interpolation function and f_up is a 3× 3 convolution layer. By repeating the function <ref>, the feature map with higher spatial resolution could be obtained. Finally, the high spectral resolution HSI could be recovered from the HSR feature map via

    X^* = f_rec( Z_h),

where f_rec is the convolution layer for mapping 96-channeled feature maps to B-channeled ones. Note that the kernel size of most convolution layers is 3× 3, except for the proposed FRDB-Net. The rest of the question is the design of the reduced IMFA and the proposed FRDB networks. More specifically, a larger kernel size used in the proposed FRDB could result in heavy computational effort, how to effectively reduce the complexity of the FRDB is essential.


Lightweight Design. In this subsection, we introduce the base blocks for IMFA-Net and FRDB-Net, which are both effective and efficient. The detailed network architectures of these blocks are depicted in Figure <ref>. As illustrated in Figure <ref>, each block in both IMFA-Net and FRDB-Net comprises three stacked base blocks with a shortcut connection linking the input and output feature maps (Figure <ref>(a)). Within each base block, the input feature map is condensed into a c_s-channeled feature map and merged with the input feature map, resulting in a c_s + N_Base-channeled feature map (Figure <ref>(b)(c)). Following DCSN <cit.>, we set N_Base to 64 and reduce c_s to 16 from 32, decreasing computational complexity.

For the FRDB-Net, we draw inspiration from <cit.> and use a residual dense block (RDB) to construct its base block. We identify that feature map reuse is the primary complexity factor in the RDB feed-forward process. Utilizing depthwise separable convolution, as in MobileNet, effectively reduces parameters and computational complexity. Additionally, grouped convolution, employed in ResNeSt, further decreases complexity with an optimal number of groups c_g. We apply grouped convolution in our FRDB base block to achieve a balance between parameter count and complexity. Specifically, we set c_g=N_Base=c_s=96, reducing the computational complexity in the last layer of the FRDB base block (Figure <ref>(b)) by a factor of 5. This approach shows that grouped convolution can lower computational complexity without compromising performance.

The 2D convolution exploits spatial domain local correlation for complexity reduction, just as the high redundancy between HSI bands permits further computational efficiency. Figure <ref> illustrates the efficacy of grouped convolution in this context. Given the higher redundancy among successive HSI bands than distant ones, grouped convolution naturally leverages this redundancy, significantly reducing computational complexity and parameter count without sacrificing performance. For example, with a 172× 128× 4 stripe-like HSI, one feature map can be derived from 43 bands (172/ 4) using grouped convolution when c_g=4, efficiently exploring local redundancy in the spectral domain. The proposed decoder, incorporating IMFA-Net, capitalizes on cross-scale feature interaction, ensuring promising performance. 



 §.§ Task-Specific Optimization

To boost efficiency, especially in the grouped convolution of our proposed FRDB-Net, we have significantly reduced the number of channels in the feature maps of the RTCS. This reduction could potentially lead to a slight decrease in performance, particularly in terms of spectral quality, due to the constrained parameter space. For instance, the proposed method comprises approximately 6.291M parameters, in contrast to the 11.951M parameters in DCSN <cit.>. DCSN <cit.> employs solely the ℓ_1 loss for training, which might not fully address the variance across different bands, reflecting diverse values. This could mean that the ℓ_1 loss might inadvertently prioritize minimizing bands with higher values, potentially compromising spectral fidelity in certain bands.

Recognizing the paramount importance of spectral index in HCS, we introduce a SAM loss function to guide the proposed RTCS towards superior spectral index outcomes more effectively. The SAM loss is defined as:
    
    ℓ_SAM = 1/N∑_n=1^Ncos^-1( clip( X_n^T X^*_n + ϵ/X_n_2 ·X_n^*_2 + ϵ) )

where X_n and X^*_n represent the n-th spectral vector of the ground truth and the reconstructed HSI, respectively. The SAM loss calculates the angular difference between these spectral vectors, using the arc-cosine of the normalized dot product to measure cosine similarity. We add a small constant ϵ to the denominator to prevent division by zero, thus enhancing numerical stability. The clip function constrains the similarity range between -1 and 1, ensuring stable arc-cosine calculations. By focusing on angular differences instead of magnitudes, this loss provides a robust metric for assessing the spectral fidelity of the reconstructed HSI compared to the original. It emphasizes the orientation of spectral signatures over their amplitude, facilitating a more accurate comparison of spectral characteristics.


    ℓ_T( X,  X^*) = ‖ X -  X^* ‖ + αℓ_SAM( X,  X^*),

where α is the weighting factor to control the balance between ℓ_1 and the proposed SAM loss. 
Furthermore, the stripe effect is often presented in the existing hyperspectral image datasets due to sensor errors or electrical exceptions. We also introduce this phenomenon into the training process via random masking on the hyperspectral stripes. In this fashion, we could enable the proposed RTCS to restore the complete hyperspectral stripes from the missing one during the decoding stage. Specifically, let the binary mask be M ∈ℝ^B× H× W, the masked HSI could be

    X_mask =  X ⊙ M.


In our proposed method, the use of a mask is crucial, yet its design remains relatively simple. The diverse characteristics of the stripe effect make it impractical to capture all possible types during the training process. To address this, we have implemented a basic mask design that involves randomly masking parts of successive bands within a set of hyperspectral stripes. Based on our research, we found that the likelihood of a hyperspectral stripe being affected by stripe effects is about 0.2, and the maximum number of consecutive bands impacted by stripe effects is also 0.2. This strategic approach effectively equips the proposed RTCS technique to reconstruct hyperspectral stripes while reducing the impact of the stripe effect. It ensures that the RTCS can effectively reconstruct hyperspectral data using only partially available compressed information.

In order to guide the proposed RTCS in effectively restoring the HSI with stripe effect, the total loss function used for data imputation could be


    ℓ_Aug.   = ℓ_T( X,  X^*) 
       + ‖ X_mask -  X^*_mask‖ + αℓ_SAM( X_mask,  X^*_mask).

Finally, the robust RTCS could be trained with the proposed total loss function ℓ_Aug.. The mask selection and design used in ℓ_Aug. are discussed in the experiments. 
		
	

§ EXPERIMENTAL RESULTS




 §.§ Experimental Settings


	
	Dataset descriptions. 
    The dataset used in this study was collected from AVIRIS <cit.> sensor, which was pre-processed in the same way as DCSN <cit.>. We collected the HSI data between 2008-2012, and the original HSI data was partitioned into 256 × 256 sub-images, resulting in 1735 HSI sub-images. We then applied the k-means to cluster the HSI sub-images into five types, including City, Mountain, Forest, Farm, and Other types, and manually corrected any misclassified HSI sub-images. We selected 200 HSI sub-images from each class, resulting in a total of 1200 HSI sub-images as the dataset. The dataset was split into training, validation, and testing sets, consisting of 1000, 100, and 100 HSI sub-images. The dataset will be released.

    The size of the stripe-like HSI is designed to be 172× 128 × 4 in the training phase, while 172× 256× 4 stripe-like HSI is used in the evaluation phase, leading to that only very limited memory usage is required for storing four hyperspectral stripes for compressed sensing. But the proposed RTCS still works well for a single hyperspectral stripe-based compressed sensing. To evaluate the performance, we stack the decoded HSI stripes to restore the HSI sub-image with 172× 256× 256 since the optimization-based approach requires full 3D-tensor as the input, followed by measuring their objective quality by the metrics in <cit.>, including Peak signal-to-noise ratio (PSNR), Root-mean-square error (RMSE), and the Spectral Angle Mapper (SAM). 
    
    
	Hyper-parameter settings.   
    
    For the training phase of our RTCS model, the AdamW optimizer <cit.>, an improved version of traditional Stochastic Gradient Descent (SGD) algorithm, is employed to train our network, where (β_1, β_2) are (0.9, 0.999), and the epsilon ϵ is 1e-8. Considering the possible diversity of HSI datasets, including exhibiting extremely high variability due to their rich spectral information and complex spatial features, the AdamW optimizer is crucial to enhance the generalization ability of the model.
    
    The initial learning rate is 1e-4 with the step-wise learning rate decaying every 1,000 epochs with scaling factor 1/2, and the batch size is set to 10. The number of total epochs of the proposed RTCS in the training phase is 5,000. The sampling rate s_r of the proposed RTCS is determined by the size of the compact representation Z of the HSI X. Specifically, the spatial size of the HSI is reduced to be 1/16 (i.e., 1/4 for width and height) for the target sampling rate s_r ≤ 1%, while the number of the feature maps b of the compressed HSI Z could be calculated based on b =⌊  B× s_r × 16 ⌋. While the sampling rate s_r≥ 1%, the spatial reduction is 1/4 and the number of channels of Z could be b =⌊  B× s_r × 4 ⌋.
















				






    
    


 §.§ Performance Evaluation

    
    To illustrate the superiority of RTCS, seven state-of-the-art HCS methods are selected as baselines, including AAHCS<cit.>, SPACE<cit.>, SpeCA<cit.>, TenTV<cit.>, Hyper-LSS (H-LSS) <cit.>, and the prior art, DCSN<cit.>. 

    We follow the standard HSI-based metrics used in DCSN <cit.>, including PSNR, RMSE, and SAM, to evaluate the performance of the proposed and other peer methods, where PSNR and RMSE are used to evaluate the spatial fidelity while SAM is adopted to verify the degree of spectral preservation of the reconstructed HSI. 



  §.§.§ Objective Quality Assessment
  

To demonstrate the robustness and generalizability of our RTCS framework, we carried out cross-type dataset evaluations. The RTCS was trained on one type from the dataset and then tested across four different types. As shown in Table <ref>, the performance of RTCS is benchmarked against other methods using metrics like SAM, PSNR, and RMSE. Across all these indices, RTCS consistently surpasses other methods, proving its robustness even when applied to a single hyperspectral stripe without relying on a large memory buffer, as evidenced by the last row in the Table <ref>.

Significantly, RTCS shows superior performance compared to other deep-learning-based HCS methods, such as DCSN <cit.>, despite being trained on a smaller dataset. While DCSN uses 2,300 training samples, RTCS requires only 1,000, highlighting the efficiency of our model in a limited-data scenario. This difference indicates that more complex network structures might need larger datasets for similar performance levels. By simplifying the network architecture, RTCS not only accommodates but excels in few-shot learning scenarios, demonstrating its practicality and efficiency.

Furthermore, as the sampling rate increases, for instance, to 5%, most HCS methods, including RTCS, AAHCS <cit.>, and DCSN <cit.>, show significant performance improvements, as seen in the Table <ref>. In AAHCS , self-similarity regularization in the spatial domain effectively aids the restoration task, yielding comparable PSNR and RMSE for the reconstructed HSI. However, RTCS outperforms AAHCS in terms of SAM, suggesting superior preservation of spectral signatures by RTCS.

    

    Consider a scenario where bandwidth is severely limited due to an increase in the number of miniaturized satellites. In such cases, a viable solution might involve reducing the sampling rate of the proposed RTCS even further. As shown in the Table <ref>, the RTCS demonstrates considerable performance at an approximate sampling rate of 0.5%, while the performance of DCSN <cit.> markedly declines at such low rates. It is worth noting that some partial optimization-based HCS methods, such as AAHCS <cit.> and SPACE <cit.>, couldn't assess performance at this low sampling rate due to significant memory usage and prolonged decoding times. Consequently, the proposed RTCS stands out, especially at lower sampling rates (s_r), achieving state-of-the-art performance compared to learning-based HCS methods like DCSN. Furthermore, optimization-based HCS methods generally struggle to deliver satisfactory results at very low sampling rates.

    When comparing the performance of the proposed RTCS with other advanced HCS methods, RTCS distinctly surpasses them. Notably, RTCS effectively manages the stripe effect and transmission noise associated with pushbroom scanning, delivering promising results. Additionally, RTCS is fast, lightweight, and hardware-friendly, making it suitable for real-time applications without compromising its leading performance. In Section <ref>, we will delve into the robustness of the proposed RTCS, encompassing aspects such as restoration with stripe effect, generalizability, and resistance to transmission noise.
    
    
    
    
    








    



 §.§ Robustness Evaluation
 
While the performance of the proposed RTCS achieves state-of-the-art in the ideal case, the robustness and stability evaluation of the proposed RTCS for real-world scenarios of the HCS and data transmission could be more essential. We conduct the robustness evaluation in this subsection, including the cases of channel noise during transmission, stripe effect due to partially and temporarily broken sensors, and few-shot compatibility.





  §.§.§ Restoration with Missing Data

As longer-running sensors at the miniaturized satellite in space, a partial spectrum could be missing during compressed sensing, known as the stripe effect of HSI. Most approach treats this could be resolved by post-processing, known as HSI inpainting techniques <cit.>. With the currently accessible data portal of HSI, some stripe-type stripe effects in the multiple successive bands could be found in <cit.>, which is also shown in the HSI inpainting approach ADMM-ADAM <cit.>). In this study, we show that the proposed RTCS can recover the complete HSI from the corrupted HSI using the proposed loss function ℓ_Aug.. 

To evaluate the robustness of the stripe effect, three types of stripe effect of HSI are studied in this experiment, as follows:

 
    
  * PM (Partially missing): Some of the lines of the HSI stripe are sometimes missing at continuous bands due to unexpected electrical errors on the miniaturized satellite, as an example shown in the Figure <ref>(a). It is the lightest missing type.
    
  * BM (Band-wise missing): The sensor for a range of the bands is partially broken, resulting in partially corrupted data in each HSI stripe, as shown in the regular pattern of Figure <ref>(b).
    
  * CM (Completely missing): The sensor for a range of the bands is completely broken, resulting in no information in the continuous bands, as shown in the Figure <ref> (c).

Only PM and CM are adopted for ℓ_Aug. with the randomly selected stripes and the continuous bands in the training phase. In the testing phase, the fixed missing pattern for three types is generated by the fixed random seed to ensure the fairness of the performance comparison between the proposed and other HCS methods. Note that the BM is not involved in the training phase; thus the results of the BM in the testing phase could be regarded as the generalizability evaluation. 


    
    
Table <ref> presents the performance comparison between the proposed RTCS and other HCS methods for three stripe effect types. Note that the sampling rate s_r in the experiments is trained and evaluated on ≈ 1%. All the stripe effects could result in significant performance degradation for most of the HCS methods, except for the proposed RTCS. An even worse case is assumed that the partial sensors for the specific bands are completely broken (i.e., CM), resulting in a discontinuity at a range in the spectral domain. In this case, the spectral signature (i.e., SAM) of the reconstructed HSI is significantly degraded for most of the HCS methods under the stripe effect. However, the proposed RTCS still remains strong, even in successive 30 bands completely missing. In BM type, the proposed RTCS also presents promising performance, implying that the proposed RTCS has the ability to deal with the new missing types which are not observed in the training phase. While the DCSN achieves promising performance on the pure HCS, the stripe effect still significantly harms the effectiveness due to the lack of the proposed ℓ_Aug. In this way, the compressed hyperspectral stripes could be well reconstructed based on the spectral and spatial context, greatly reducing the stripe effect of the reconstructed HSI based on the decoded and restored hyperspectral stripes. Figure <ref> reveals the visualized results using the false color of the reconstructed HSI based on the proposed RTCS and other peer methods in CM-type distortion, showing that the proposed RTCS presents more reliable and high-spectral-spatial quality of the reconstructed HSI under stripe effect scenarios.



As for visualized results comparison, the false RGB image of HSI could be synthesized by normalizing the band indices 13, 25, and 61. Figure <ref> illustrates the false RGB image visualized from the reconstructed HSI of the different HCS methods under the 50-80 bands completely missing (i.e., CM), in which the red spectrum is also missing. As a result, the visualized results of the proposed RTCS are significantly better than that of other peer methods, implying the spectral signature of the reconstructed HSI using the proposed RTCS is promising. The state-of-the-art HCS methods, AAHCS <cit.> and DCSN <cit.>, fail to reconstruct the visually pleasing results, implying the stripe effect could greatly influence the compressed signal Z. Although the state-of-the-art inpainting method is adopted to recover the complete HSI from the corrupted one, the lower spatial and spectral quality cannot provide rich enough information for further remote-sensing-related analysis. Since the results reported in ADMM-ADAM <cit.> is evaluated in the pure and high-quality HSI with stripe effect, it is hard to guarantee such high-quality of the decoded HSI under a very low sampling rate (e.g., ≈ 1%), leading to unstable performance. In contrast, the proposed RTCS successfully reconstructs the missing parts from the compressed representation directly, leading to a promising visual quality of the decoded HSI and enabling the other downstream applications possible, even when some sensors are partly/completely broken.

As one can argue that the missing types could vary due to different types of sensors, the proposed RTCS requires to re-train the network, consisting of encoder and decoder, to enable the capability of HSI inpainting of the new missing type. Notably, the proposed RTCS is few-shot-compatible (cf. the following section), leading the cost of fine-tuning the RTCS for new types of stripe effects could be low and easy.    


 

	





  §.§.§ Generalizability Evaluation

The data-driven approach usually suffers performance degradation issues from insufficient training samples. Compared to the optimization-based HCS methods, the data collection in the DL-based HCS methods is usually tedious and time-consuming. Since the number of parameters of the proposed RTCS is relatively small, the proposed RTCS should be able to train well using a relatively small-scale training set without overfitting. In this Section, we reduce the number of training data by randomly sampling 10% to 40% from the original training set (say, 1,000 HSI) to evaluate the performance of the same test set (say, 100 HSI). As shown in the Table <ref>, the proposed RTCS could obtain the promising performance using only 10% training samples, i.e., 100 HSI only, while the DCSN <cit.> requires 100% training set to have similar performance. As claimed in the ADMM-ADAM theory <cit.>, a huge amount of the HSI collection is infeasible, so they have developed ADMM-ADAM using just small training data consisting of 400 HSI. However, we show that the proposed RTCS is more effective, even with only 100 HSI as the training set. As shown in the Figure <ref>, the proposed RTCS improves smoothly when the number of training samples increases to the 30% of the original dataset, implying that the performance of our RTCS becomes promising using just a small amount of the training data. 

Besides data collection is time-consuming for DL-based HCS methods, however, the proposed RTCS only requires a tiny amount of the training samples, making our RTCS useful and practical. Once a new type of HSI sensor appears, the proposed RTCS requires only small data, making the HCS training and deployment more effective and efficient. 


Our RTCS was developed on a proprietary dataset in the initial phase, laying the groundwork for a foundational model in hyperspectral imaging (HSI) compressed sensing. This foundational model facilitates rapid domain adaptation through fine-tuning with a limited subset of a target dataset—a method that proves to be computationally economical.

To evaluate the framework's performance, we engaged in further experimentation using an auxiliary dataset referenced as <cit.>. The RTCS framework was subjected to training on the designated training subset of this dataset, with its effectiveness subsequently gauged on the test subset. In an endeavor to validate the model's applicability, only 10% of the <cit.> training samples were employed to fine-tune the pre-established RTCS model, which substantiated its efficacy on the test subset. The DCSN dataset <cit.> encompasses 2300 HSI samples, encapsulating a variety of landscapes, including City (C-type), Land (L-type), Farm (F-type), and Mountainous (M-type) terrains. Training the RTCS on the entirety of this dataset is projected to enhance performance; however, the collection of comprehensive datasets for novel target HSI applications is often constrained by practicality and resource availability.

The few-shot compatibility embedded within our RTCS enables the efficient training of the network on novel datasets with a minimal quantity of samples, mirroring the process of task-specific fine-tuning seen with foundational models. Illustrated in the Table <ref>, the RTCS achieves state-of-the-art performance with a mere 1% of the training samples, equivalent to 23 HSI images. This advancement highlights the significance and viability of Few-shot Compatibility in practical settings, where extensive data collection and model training are typically arduous. Crucially, this feature permits the swift and efficacious deployment of our RTCS across a spectrum of HSI types, employing only a handful of training samples and thereby promoting extensive adaptability and practicality for CubeSat HSI applications.




    





    








  §.§.§ Transmission Noise


In order to evaluate the robustness of the proposed method, we follow the training process suggested in Denoised AutoEncoder<cit.> and DCSN <cit.> to add the Gaussian noise with the 30dB Signal-to-Noise-Ratio (SNR) to learn the robust RTCS model. The rest hyperparameters of the proposed RTCS are identical to the default settings. Table <ref> draws the performance comparison between the proposed RTCS and other state-of-the-art HCS methods under sampling rates ≈ 1% and ≈ 5%. As a result, the proposed RTCS significantly outperforms the other HCS methods in terms of SAM, PSNR, and RMSE. Remarkably, the Robust DCSN (RDCSN) <cit.> also adopts the Gaussian noise during the training phase to make the model could robust to noise. However, the proposed RTCS still achieves a notable performance improvement, implying that the architecture of the proposed RTCS is better than that of DCSN <cit.> naturally. The results in the Figure <ref> has shown the efficacy of the proposed RTCS. Compared with other HCS methods that have varying degrees of chromatic aberration, the reconstruction results of RTCS are closer to the ground-truth.




 §.§ Computational Complexity Analysis

We compare the proposed RTCS to the other state-of-the-art HCS methods for computational complexity and run-time of encoding and decoding. Note that each result is measured with the same hardware. In the general computational comparison, a personal computer equipped with an i9-9900 CPU, 128GB Memory, and a low-end graphics processing unit (GPU) is used to measure the run-time of HCS for an HSI sized of 172× 256× 256. Although the proposed RTCS is a stripe-like HCS, the run-time of the RTCS and DCSN <cit.> is measured in the whole HSI sized of 172× 256× 256 since the optimization-based HCS methods work for the whole HSI only. In order to verify the efficiency of the proposed RTCS, we adopt NVIDIA Jeston TX2 to measure the run-time of the proposed RTCS and DCSN <cit.>. Note that the optimization-based HCS methods were developed based on Matlab, which is incompatible with ARM-based architecture; thus, those methods cannot run in Jetson TX2. We take the average of the run-time by repeatedly executing the inference 30 times. 

In the Table <ref>, the proposed RTCS has the lowest parameter space as well as the computational complexity compared to other optimization- and DL-based approaches. In the run-time comparison, the proposed RTCS achieves the fastest inference time since the depth of the network is relatively shallow and the number of parameters is low, making real-time compressed sensing and decoding possible. Although AAHCS <cit.> achieves good performance as illustrated in Section <ref> under higher sampling rate s_r ≈ 5%, the decoding process of the proposed RTCS is 3000x faster than that of AAHCS <cit.>, implying our RTCS is more feasible for real-time hyperspectral stripes decoding. Moreover, with the sampling rate ≈ 1%, the proposed RTCS significantly outperforms the peer methods for all quality indices, as well as achieves the lowest computational complexity. On the one hand, the encoder of the proposed RTCS only requires a matrix multiplication, making the cost of the computation and memory usage could be minimized in the miniaturized satellite. On the other hand, the decoder of the proposed RTCS could be deployed on edge devices due to its computational cheapness, which could greatly reduce the overhead of the central server and make more critical HSI applications possible.

More specifically, due to the real-time decoding requirement of the proposed RTCS, we deploy the proposed RTCS to the low-end device, NVIDIA Jetson TX2, with the TVM compiler <cit.> for optimizing the efficiency of the DL-based approach (i.e., DCSN and our RTCS) for the specific hardware. Clearly, the proposed RTCS still achieves a very fast inference in the embedding platform, i.e., 19 Milliseconds (ms), while the DCSN in TX2 requires 25 ms for inference. In the DL-based HCS methods without TVM-optimization, the proposed RTCS still meets the real-time requirements (say, 1/0.024 = 41.6 FPS), while DCSN is relatively hard to achieve such a fast inference process due to deeper network. Even in the ARM-based CPU in TX2, the inference time of the proposed RTCS is only 2 seconds, whereas the optimization-based HCS methods, including AAHCS <cit.>, SpeCA <cit.>, and SPACE <cit.>, failed to decode the HSI due to limited memory and insufficient computing power. Therefore, we could conduct that the proposed RTCS is efficient, as well as a state-of-the-art HCS method for miniaturized satellites.

We also further verify that the proposed RTCS is hardware friendly. The proposed encoder could deploy the HCS in the lightweight chip without any hardware accelerator, such as a GPU or intelligent processing unit (IPU), in a real-time sense. Although only one matrix multiplication is required in the encoder of the proposed RTCS, the floating-point with 32-bit precision (FP32) is still required, the same as the optimization-based HCS methods. Most low-end computing chips, such as Kneron KL720/FL520, require integer-based inference to maximize efficiency. Therefore, we further study the model quantization for our encoder to enable integer computation for saving power-consuming purposes by using quantization-aware training (QAT) and post-quantization (PQ) <cit.>. As shown in the Table <ref>, the quantized encoder of RTCS maintains promising performance while being more hardware-friendly."



Compared to the traditional optimization-based approach, the floating point operation is strictly required in the miniaturized satellite, leading to a higher computational cost inside the computational chip. As claimed in AAHCS <cit.>, the basis could be calculated first and only all-addition operations required in the miniaturized satellite. However, the performance is unknown since the experiments were conducted on a data-dependant basis. In this fashion, the basis decomposition is still needed, so a number of floating point operations are required, which is computationally expensive. As a result, the encoder of the proposed RTCS could be state-of-the-art as well as computationally cheap.


 
	
	
	
	
	
	
	
	
	
	
	
 
	
	
	
 

     	



 §.§ Ablation Study

The proposed RTCS primarily consists of two distinct branches designed to aggregate features with varying equivalent receptive fields. In order to assess the efficacy of the dual-stream network architecture, we examine the performance of each individual stream within the proposed RTCS. We increase the number of base blocks, denoted as n_f, to 16 in order to bring the parameter space of each branch closer to that of the proposed RTCS. Table <ref> presents an ablation study of the proposed RTCS to validate the effectiveness of both the two-streamed architecture and the SAM loss.

In the first stream, the IMFA-Net-based HCS, which can be considered an enhanced version of DCSN <cit.>, demonstrates a high PSNR but limited spectral quality, as indicated by the SAM value. Comparing IMFA-Net with the original design of DCSN <cit.>, it becomes evident that more channels in the IMFA-Net decoder are necessary to restore the high-quality spectral signature from compressed HSI effectively. When the SAM loss is considered during the training phase, spectral quality is further improved, suggesting that the proposed SAM loss is better suited for preserving HSI signatures in HCS tasks. Owing to the simplified network of IMFA-Net, its performance surpasses that of DCSN, mainly when dealing with a small training set.

Likewise, the performance of the FRDB-Net-based HCS can be enhanced when both loss functions are utilized concurrently. In the proposed dual-stream architecture, significant performance improvements are achieved as a result of the judicious aggregation of multiple receptive fields. Although the number of convolutional kernels is considerably reduced through the use of grouped convolution, leading to diminished performance in the HSI spectral signature, the proposed SAM loss can guide network learning toward spectral preservation, resulting in a superior SAM index for the proposed RTCS.

In conclusion, the proposed dual-stream network architecture combined with the SAM loss demonstrates effectiveness, efficiency, robustness, and hardware compatibility, making it a valuable solution for the task at hand.





    

	
	

§ CONCLUSION

This study proposed an effective, efficient, and robust hyperspectral compressed sensing (HCS) network, termed real-time compressed sensing (RTCS), for significantly lower computational resources required in the encoder side at the miniaturized satellite, as well as hardware-friendly decoder inside the receiver based on edge devices.
  
  A very computationally cheap encoder with only one matrix multiplication with integer-8 precision, has been proposed to compressively sense the stripe-like hyperspectral image (HSI),  making HCS on CubSat possible and practical.
  
  A two-streamed-based decoder, consisting of two shallow and effective sub-networks, has been proposed to reduce the computational complexity to meet the requirements of edge devices by judiciously aggregating the middle-level and semantic features by the larger and regular convolutional kernel sizes for two sub-networks.
  

Benefiting from the lower number of parameters in the proposed RTCS, the proposed method only requires small data, making its high adaptation power for new datasets.
  
  A novel task-specific training policy with Spectral Angle Mapper-aware (SAM) loss has been proposed to improve the quality of the spectral signature preservation and enable the proposed RTCS to restore the complete HSI even if partial information of the HSI is missing or noisy. 
  
  Comprehensive experimental results demonstrated the superiority of the proposed RTCS. Moreover, the proposed RTCS is the first research to tackle the joint issue of the HSI restoration and compressed sensing in edge devices only, leading to our RTCS being practical and useful in real-world HCS applications.
  



	
	IEEEtran
		
	

[
    < g r a p h i c s >
]Chih-Chung Hsu
		
    (S'11-M'14-SM'20) received his B.S. degree in Information Management from Ling-Tung University of Science and Technology, Taiwan, in 2004, and his M.S. and Ph.D. degrees in Electrical Engineering from National Yunlin University of Science and Technology and National Tsing Hua University (NTHU), Taiwan, in 2007 and 2014, respectively.
    
    From 2014 to 2017, Dr. Hsu was a postdoctoral researcher with the Institute of Communications Engineering at NTHU. He served as an assistant professor with the Department of Management Information Systems at National Pingtung University of Science and Technology from February 2018 to 2021 and has been affiliated with the Institute of Data Science at National Cheng Kung University since 2021. His research interests primarily focus on computer vision and machine/deep learning, with applications in image and video processing. He has published several papers in top-tier journals and conference proceedings, including IEEE TPAMI, IEEE TIP, IEEE TMM, IEEE TGRS, ACM MM, IEEE ICIP, IEEE IGARSS, and IEEE ICASSP, including the best student paper award from the IEEE ICIP in 2019.
    
    Dr. Hsu became a Senior Member of the Institute of Electrical and Electronics Engineers (IEEE) in October 2020. He received the Best Young Professional Member Award from IEEE Tainan Section in 2023. Dr. Hsu leads the Advanced Computer Vision Lab (ACVLab), winning over 20 grand challenges at top-tier conferences over the years, such as the 3rd place award at the Learning to Drive Challenge from the IEEE International Conference on Computer Vision (ICCV), where he was also an invited speaker. He secured the 3rd place award in the Visual Inductive Priors for Data-Efficient Computer Vision Challenge and 1st place in the COV19D challenge at the European Conference on Computer Vision (ECCV) in 2020 and 2022. 
    


[
    < g r a p h i c s >
]Chih-Yu Jian
    earned a bachelor's degree and graduated in 2018 from the Department of Information Management at Pingtung University of Science and Technology. He obtained his master's degree from the same department in 2020.
    
    From September 2020 to November 2022, he served as a master's researcher at Chang Gung Medical Foundation Hospital. Since December 2022, he has been working as a master's researcher at Far Eastern Memorial Hospital. His main interests are focused on computer vision and machine/deep learning, as well as the applications of medical imaging and natural language processing. He has published papers in IEEE ICASSP, IEEE VCIP, and NeuroImage Clinical.
    
    Over the years, he has won several significant challenges with his team at top conferences, such as securing third place in the Learning to Drive Challenge at the IEEE International Conference on Computer Vision (ICCV) and first place in the COV19D Challenge.		



[
    < g r a p h i c s >
]Eng-Shen (James) Tu
is a research assistant in the Software Engineering and Intelligent Test Automation Lab at National Cheng Kung University in Tainan, Taiwan. Currently, he is working towards his Bachelor's degree in Computer Science and Information Engineering at the same university. Though still early in his academic journey, James continues to explore and deepen his understanding of his chosen fields, always keen to contribute to new knowledge and innovative solutions in the domain of Data Science and Software Engineering.
		



[
    < g r a p h i c s >
]Chia-Ming Lee
received his B.S. degree at the Department of Statistics and Information Science, Fu Jen Catholic University (FJCU), Taiwan, in 2023. He is currently a M.S. student with the Advanced Computer Vision Laboratory (ACVLab) at the Institute of Data Science, National Cheng Kung University (NCKU), Tainan, Taiwan. His research interests mainly focus on computer vision, deep learning and their application.
  
    He have awarded the Jury Prize at Visual Inductive Priors Workshop from the IEEE International Conference on Computer Vision (ICCV). He secured the 1st place in the AI-enabled Medical Image Analysis Workshop and COVID-19 Diagnosis Competition held at the IEEE International Conference on Acoustics, Speech, Signal Processing (ICASSP). He received the Top Paper Award in the Social Media Prediction Challenge at ACM Multimedia (ACMMM).

	
[
    < g r a p h i c s >
]Guan-Lin Chen
received the B.S. degree from the Department of Financial Engineering and Actuarial Mathematics, Soochow University, Taipei, Taiwan, in 2020, and the M.S. degree from the Institute of Data Science, National Cheng Kung University, in 2022. He currently serves as an AI engineer at MobileDrive, Taipei, focusing on the application of learning-based methods for perception cameras. His research interests primarily focus on computer vision, machine learning, and deep learning, especially explainable and trustworthy AI.


	

