abstract: **Significance:** 2D fully convolutional neural networks have been shown capable of producing maps of sO$_2$ from 2D simulated images of simple tissue models. However, their potential to produce accurate estimates *in vivo* is uncertain as they are limited by the 2D nature of the training data when the problem is inherently 3D, and they have not been tested with realistic images.
  .
  **Aim:** To demonstrate the capability of deep neural networks to process whole 3D images and output 3D maps of vascular sO$_2$ from realistic tissue models/images.
  .
  **Approach:** Two separate fully convolutional neural networks were trained to produce 3D maps of vascular blood oxygen saturation and vessel positions from multiwavelength simulated images of tissue models.
  .
  **Results:** The mean of the absolute difference between the true mean vessel sO$_2$ and the network output for 40 examples was 4.4% and the standard deviation was 4.5%.
  .
  **Conclusions:** 3D fully convolutional networks were shown capable of producing accurate sO$_2$ maps using the full extent of spatial information contained within 3D images generated under conditions mimicking real imaging scenarios. This work demonstrates that networks can cope with some of the confounding effects present in real images such as limited-view artefacts, and have the potential to produce accurate estimates *in vivo*.
author: Ciaran Bench; Andreas Hauptmann; Ben Cox
title: Towards accurate quantitative photoacoustic imaging: learning vascular blood oxygen saturation in 3D

**\***Ciaran Bench,

```latex
\begin{spacing}{1}   % use double spacing for rest of manuscript

\section{Introduction}
\label{sect:intro}  % \label{} allows reference to this section
Blood oxygen saturation (sO$_2$) is an important physiological indicator of tissue function and pathology. Often, the distribution of oxygen saturation values within a tissue is of clinical interest, and therefore, there is a demand for an imaging modality that can provide high resolution images of sO$_2$. For example, there is a known link between poor oxygenation in solid tumor cores and their resistance to chemotherapies, thus images of tumor blood oxygen saturation could be used to help stage cancers and monitor tumor therapies \cite{tomaszewski2017oxygen,ron2019volumetric}. Some imaging modalities have been shown capable of providing limited information about or related to sO$_2$ in tissue. Blood Oxygenation Level Dependent Magnetic Resonance Imaging (BOLD MRI), which is sensitive to changes in both blood volume and venous deoxyhaemoglobin concentration, can be used to image brain activity, but can not respond to changes in oxygen saturation \cite{ogawa1990brain}. Purely optical techniques, such as near-infrared spectroscopy (NIRS) and diffuse optical tomography (DOT), can be used to generate images of oxygen saturation \cite{villringer1993near,gibson2005recent}. However, because of high optical scattering in tissue, these modalities can only generate images with low spatial resolution beyond superficial depths.
%Photoacoustic tomography is capable of providing images of sO$_2$ at greater depths for a couple of reasons. Firstly,

Photoacoustic (PA) imaging is a hybrid modality that can be used to generate high resolution images of vessels and tissue at greater imaging depths than purely optical modalities \cite{beard2011biomedical}. PA image contrast depends on the optical absorption of the sample, so images of well-perfused tissues and vessels can in principle be used to generate images of sO$_2$ with high specificity. However, unlike strictly optical techniques, information about the contrast in PA images is carried by acoustic waves that can propagate from deep within a tissue to its surface undergoing little scattering. %Therefore, photoacoustic tomography can be used to generate high resolution images of vessels and tissue at greater imaging depths (and in principle generate images of the sO$_2$ of targets at significant depths, such as tumours or kidneys).

 %Information about the optical absorption coefficient of a blood sample at multiple wavelengths can be used to estimate its sO$_2$ via spectroscopic inversion (also referred to as linear unmixing) \cite{cox2012quantitative}.
In the ideal case of a perfect acoustic reconstruction, the amplitude of a voxel in a PA image can be described by
\begin{equation}
p_0(\textbf{x},\lambda) = \mu_a(\textbf{x},\lambda)\Gamma(\textbf{x})\Phi(\textbf{x},\lambda; \mu_a, \mu_s, g),
\end{equation}
where $\textbf{x}$ is the voxel's location within the sample, $\lambda$ is the optical wavelength, $\mu_a$ is the optical absorption coefficient, $\mu_s$ is the scattering coefficient, g is the optical anisotropy factor, $\Gamma$ is the PA efficiency (assumed here to be wavelength independent), and $\Phi$ is the light fluence. Images of sO$_2$ may only be recovered if the sample's absorption coefficients (or at least the absorption coefficient scaled by some wavelength independent constant, such as $\mu_a(\textbf{x},\lambda)\Gamma(\textbf{x})$) can be extracted from each image. In the hypothetical case where the sample's fluence distribution is constant with wavelength, a set of PA images acquired at multiple wavelengths automatically satisfy this requirement. However, because the optical properties of common tissue constituents are wavelength dependent, this condition is never met in \textit{in vivo} imaging scenarios \cite{hochuli2019estimating}. In general, knowledge of the fluence distribution throughout the sample at each excitation wavelength is required to accurately image sO$_2$ \cite{cox2012quantitative}. If an accurate fluence estimate is available, then an image of the sample's relative optical absorption coefficient at a particular wavelength can be obtained by performing a voxel-wise division of the image by the corresponding fluence distribution, as described by
\begin{equation}
\frac{p_0(\textbf{x},\lambda)}{\Phi(\textbf{x},\lambda)} = \mu_a(\textbf{x},\lambda)\Gamma(\textbf{x}).
\end{equation}

%Accurate estimates of the fluence distribution are difficult to acquire of the optical properties distributed throughout the entire sample is required to accurately estimate the fluence distribution in tissue.
%Tissue is highly optically scattering, and photons will be scattered and interact with various different regions within the tissue before they are absorbed. The paths that photons take as they propagate within the tissue will therefore depend on the distribution of scatterers and absorbers in the sample. The fluence is the amount of light energy passing through some unit area in any direction in a tissue. Therefore, the fluence at a point in the sample will also depend on the distribution of absorbers and scatterers within the whole tissue volume.
In some cases it might be possible to measure an estimate of the fluence using an adjunct modality \cite{hussain2016quantitative}, but more commonly, attempts have been made to model the fluence. However, because the optical properties of a tissue sample are usually not known before imaging (the only reason the fluence is estimated at all is so that unknown information about the sample's optical absorption coefficient can be recovered from the image data), it is difficult to model the fluence distribution. A variety of techniques have been developed to recover tissue absorption coefficients from PA images without total prior knowledge of the tissue's optical properties. Progress towards solving this problem can be summarized into three key phases. In the first phase, 1D analytical fluence models were used to estimate the fluence by taking advantage of assumed prior knowledge of some of the sample's optical properties, or by extracting the optical properties of the most superficial layers from image data \cite{carome1964generation,cross1987time,cross1988ablative,guo2010calibration,Deng2016,Kim2011}. In the latter case, the effective attenuation coefficient of the most superficial tissue layer (assumed to be optically homogeneous) is usually estimated by fitting an exponential curve to the decay profile of the image amplitude above the region of interest (e.g. a blood vessel).

In the next phase, sample optical properties were recovered using iterative error minimisation approaches \cite{fonseca2017three,cox2006two,buchmann2020quantitative,buchmann2019three}. With these techniques, knowledge of the underlying physics is used to formulate a model of image generation. The set of model parameters (which might include the concentrations of deoxyhaemoglobin and oxyhaemoglobin in each voxel) that minimizes the error between the images generated by the model and the experimentally acquired images are treated as estimates of the same parameters in the real images. This technique is only effective when the model of image generation is able to generate a set of simulated images very similar to the real set of images when the correct values for the chromophore concentrations are estimated. This is only possible when the image generation model is able to accurately model image acquisition in the real system. In practice, accurate models of image generation are challenging to formulate as not all aspects of the data acquisition pathway are fully characterized. Therefore, this technique has not yet been shown to be a consistently accurate method for imaging sO$_2$ in tissue. Both iterative error-minimisation and analytical techniques may require significant \textit{a priori} knowledge of sample properties, such as all the different constituent chromophore types. This information is not always available when imaging tissues \textit{in vivo}, and thus this requirement further reduces their viability as techniques for estimating sO$_2$ in realistic imaging scenarios.
%Response to Ben's comment: For LU, components of blood are needed. However, for
The recent emergence of a third phase has introduced data-driven approaches to solving the problem \cite{grohl2019estimation,yang2019eda,luke2019net,durairaj2020unsupervised,chen2020deep,yang2019quantitative, kirchner2018context, grohl2018confidence, cai2018end}. With these approaches, generic models are trained to output images of sO$_2$ or optical properties by processing a set of examples \cite{arridge2019solving}. These data-driven models find solutions without significant \textit{a priori} knowledge of sample properties, and do not require the formulation of an image generation model using assumed prior knowledge of all the aspects related to image acquisition. Techniques based on data-driven models, such as Deep Learning, have been used to estimate sO$_2$ from 2D PA images of simulated phantoms and tissue models \cite{grohl2019estimation,yang2019eda,luke2019net,chen2020deep,yang2019quantitative,cai2018end}. Fully-connected feed-forward neural networks have been trained to estimate the sO$_2$ in individual image pixels given their PA amplitude at multiple wavelengths \cite{grohl2019estimation}. Because the fluence depends on the 3D distribution of absorbers and scatters, a pixel-wise approach does not use the full information available in an image. Encoder-decoder type networks, capable of utilising spatial as well as spectral information, have been trained to process whole multiwavelength 2D images of 2D tissue models \cite{luke2019net,chen2020deep,yang2019quantitative,cai2018end}, or 2D images sliced from more realistic 3D tissue models featuring reconstruction artefacts \cite{yang2019eda}, and output a corresponding 2D image of the sO$_2$/optical absorption coefficient distribution. Although 2D convolutional neural networks can take advantage of spatial information to improve estimates of sO$_2$, networks trained on 2D images sliced from 3D images are missing information contained in other image slices that might improve their ability to learn a fluence correction. Therefore, it is important to show that networks can take advantage of all four dimensions of information from a multiwavelength PA image dataset to estimate sO$_2$. In addition to supervised learning, an unsupervised learning approach has been used to identify regions containing specific chromophores (such as oxyhaemoglobin and deoxyhaemoglobin) in 2D simulated images \cite{durairaj2020unsupervised}. The technique has not yet been used to estimate sO$_2$, and has only been tested on a single simulated phantom lacking a complex distribution of absorbers and scatterers that would normally be found in \textit{in vivo} imaging scenarios. %Therefore, the technique's ability to estimate sO$_2$ from more realistic tissue models is at best uncertain.

%kirchner is not deep learning, and relies on homogenous scattering

As we aim towards developing a technique for estimating 3D sO$_2$ distributions from \textit{in vivo} image data, a more robust demonstration of a data-driven technique's ability to acquire accurate sO$_2$ estimates by processing whole 3D images of realistic tissue models is desired.
We trained two encoder-decoder type networks with skip connections to 1) output a 3D image of vascular sO$_2$, and 2) output an image of vessel locations from multiwavelength (784 nm, 796 nm, 808 nm, 820 nm) images of realistic vascular architectures immersed in three-layer skin models, featuring noise, and reconstruction artefacts. Details about how the simulated images were generated are described in Section \ref{sec:Image_gen}. Section \ref{sec:Traininng_param} describes the network architecture and details about the training process. Section \ref{sec:Results} describes the results.

\section{Generating Simulated Images}
\label{sec:Image_gen}
Ideally, a network trained to estimate sO$_2$ from \textit{in vivo} images would be capable of generating accurate estimates from a wide range of tissue samples with varying optical properties and distributions of vessels. Additionally, the network should be able to do this despite the presence of reconstruction artefacts and noise. This section describes each step involved in the generation of the simulated images used in this study.

\subsection{Tissue Models}
A set of several hundred tissue models, each featuring a unique vascular architecture and distribution of optical properties, were generated by immersing 3D vessel models acquired from CT images of human lung vessels into 3D, three-layer skin models (some examples are shown in Fig. \ref{fig:Vessel_models}) \cite{via_group,hauptmann2018model}. Each skin model consisted of an epidermis, dermis and hypodermis layer. The thickness of each skin layer (epidermis: 0.1 - 0.3 mm , dermis: 1.3 mm - 2.9 mm, hypodermis: 0.8 mm - 2.6 mm), and the optical absorption properties of the epidermis and dermis layers were varied for each tissue model. A unique tissue model was generated for each vascular model. The equations used to calculate the optical properties of each skin layer and the vessels at each excitation wavelength (784 nm, 796 nm, 808 nm, 820 nm) are presented in Table \ref{table:Skinproperties} in Appendix \ref{sec:Optical_properties}. These wavelengths were chosen as they fell within the NIR and data was available for all skin layers at these wavelengths. The absorption properties of the epidermis layer of each tissue model were determined by choosing a random value for the melanosome volume fraction that was within expected the physiological range. The absorption properties of the dermis layer were determined by choosing random values for the blood volume fraction and dermis blood sO$_2$ within the expected physiological range. For each tissue model, each independent vascular body was randomly assigned one of three randomly generated sO$_2$ values between $0\%$ and $100\%$. The PA efficiency throughout the tissue was set to one with no loss of generality. %In reality, the PA efficiency will vary between tissue types, and have values other than one. The PA efficiency is often set to one (or at least some constant value) in tissue models used for \textit{in silico} studies of sO$_2$ estimates derived from PA image data, as the wavelength independent constant does not affect the accuracy of linear unmixing techniques used to recover ratiometric quantities such as sO$_2$. However, it is currently unclear as to whether the PA efficiency might negatively impact sO$_2$ estimates acquired using a convolutional neural network. Eventually, this will have to be addressed if the technique is to be used with images acquired \textit{in vivo}.

\begin{figure}[tbp]
\centering
%\includegraphics[width=.97\columnwidth]{Seg_lung_mask.png}
\includegraphics[width=.6\columnwidth]{Vessel_tissue_model.png}
\caption{Left: Example of a 3D vessel model (acquired from CT images of human lungs) used to construct 3D tissue models. Right: Schematic of three-layer skin model used to construct tissue models.}
\label{fig:Vessel_models}
\end{figure}

\subsection{Fluence Simulations}
The fluence in each tissue model at each excitation wavelength was simulated with MCXLAB, a MATLAB package that implements a Monte Carlo (MC) model of light transport (considered the gold standard for estimating the fluence distribution in tissue models) \cite{fang2009monte}. Fluence simulations were run with 10$^{9}$ photons, the maximum number of photons that could be used to generate 1024 sets of images in approximately one week using a single NVIDIA Titan X Maxwell GPU with 3072 CUDA cores and 12 GB of memory. A large number of photons were used in order to reduce the MC variance to the point where it no longer contributed significantly to the noise in the simulated data. Noise was subsequently added in a systematic way to the simulated time series, as described in Section \ref{sec:Acoustic}.

MC simulations were run with voxel sidelengths of 0.1 mm, and simulation volumes with dimensions of 40 x 120 x 120 voxels. The fluence was calculated from the flux output from MCXLAB by integrating over time using timesteps of 0.01 ns for a total of 1 ns, which was sufficient to capture the contributions from the vast majority of the scattered photons. A truncated Gaussian beam with a waist radius of 140 voxels, with its centre placed on the centre of the top layer of epidermis tissue was used as the excitation source for this simulation. Photons exiting the domain were terminated. The fluence simulations were not scaled by any real unit of energy, as images were normalized before inserting them into the network. Each fluence distribution was multiplied pixel-wise by an image of the tissue model's corresponding optical absorption coefficients to produce images of the initial pressure distribution at each excitation wavelength.

\subsection{Acoustic Propagation and Image Reconstruction}
\label{sec:Acoustic}
Simulations of the acoustic propagation of the initial pressure distributions from each tissue model, the detection of the corresponding acoustic pressure time series at the tissue surface by a detector with a planar geometry, and the time reversal reconstruction of the initial pressure distributions from these times series were executed in k-Wave \cite{treeby2010k}. Simulations were designed with a grid spacing of 0.1 mm, dimensions of 40 x 120 x 120 voxels, and a perfectly matched layer of 10 voxels surrounding the simulation environment. Each tissue model was assigned a homogeneous sound speed of 1500 m/s. 2D planar sensor arrays are often used to image tissue \textit{in vivo}, as it is a convenient geometry for accessing various regions on the body \cite{plumb2018rapid,huynh2016photoacoustic,huynh2017sub}. A sensor array with a planar geometry was used in this study to mimic conditions expected in real imaging scenarios. A 2D planar sensor mask covering the top plane of the tissue model was used to acquire the time series data. Because of its limited-view geometry, the sensor array will detect less pressure data emitted from deeper within the tissue, as these regions will subtend a smaller angle with the sensor. As a result, the reconstruction will have limited-view artefacts which will become more pronounced with depth \cite{xu2004reconstructions}.

To avoid the large grid dimensions that would be required to capture the abrupt change in the acoustic pressure distribution at the tissue surface, and consequently the long simulation times, the background signal in the top three voxel planes was set to zero. This has a similar effect to the bandlimiting of the signal during measurement that would occur in practice, and has no effect on the simulation of the artefacts around the vessels due to the limited detection aperture. Furthermore, in experimental images the superficial layer is often stripped away to aid the visualisation of the underlying structures. Similar approaches have been used to improve sO$_2$ estimates generated by 2D networks. In \cite{luke2019net}, the ten most superficial pixel rows were removed from images before training to ensure that features deeper within the tissue (and therefore dimmer than the comparatively bright superficial layers) were more detectable. Similarly, superficial voxel layers were removed from images in \cite{yang2019eda} to improve the accuracy of sO$_2$ estimates.

%In k-Wave simulations, values for the acoustic variables between grid points are interpolated using a Fourier series. At discontinuities in the field, the Fourier series will oscillate (a Gibb's phenomenon). The acoustic field of the epidermis region of the tissue has a large discontinuity with the border of the image, and this introduces oscillations in the Fourier series of the acoustic field, and hence artefacts in images reconstructed from this data \cite{robertson2017accurate}. Because there is a large discontinuity between the sensor and the highly absorbing epidermis skin layer, the top three voxel planes in each initial pressure distribution were set to zero before simulating their acoustic propagation. By zeroing the epidermis region of the tissue model, the discontinuity in the acoustic field was made less severe (as the intensity of the image in the dermis regions was much lower), and bar-like artefacts in the reconstruction resulting from the Gibb's phenomenon were no longer present. A finer grid spacing in the epidermis region could be used to reduce the severity of the artefacts without this zeroing step, as this would enable higher frequencies in the Fourier series to be captured. However, because it is not very important for this work, and because it was more time efficient to simply remove the epidermis region of the initial pressure before reconstruction, this was not performed (the top layers of an image are often removed for visualisation purposes anyway).

Noise was added to each datapoint in the simulated pressure time series by adding a random number sampled from a Gaussian distribution with a standard deviation of 1\% of the maximum value over all time series data generated from the same image, resulting in realistic SNRs of about 21 dB. Details of how this noise test was carried out, and how the SNR was calculated are provided in Appendix \ref{sec:Noise_test}.

\section{Network Architecture and Training Parameters}
\label{sec:Traininng_param}
A convolutional encoder-decoder type network with skip connections (EDS) (shown in Fig. \ref{fig:architecture}, and denoted as network $A$) was trained to output an image of the sO$_2$ distribution in each tissue model from 3D image data acquired at four wavelengths. Another network, network $B$, was assigned an identical architecture to network $A$ and was trained to output an image of vessel locations from the image sets (thereby segmenting the vessels). An EDS architecture was chosen for each task, as they have been shown to perform well at image-to-image regression tasks (i.e. tasks where the input data is a set of images, and the output is an image) \cite{ronneberger2015u}. The architecture takes reconstructed 3D images of a tissue model acquired at each excitation wavelength as an input. The multi-scale nature of the network allows it to capture information about features at various resolutions, and use image context at multiple scales \cite{piao2019accuracy,zeiler2013stochastic,jaderberg2015spatial}. The network's skip connections improve the stability of training, and help retain information at finer resolutions. Finally, the network outputs a single 3D feature map of the sO$_2$ distribution, or the vessel segmentation map.

%(it can use information about how features may be distributed in the original image at high resolution, as well as more detailed information about specific features in lower spatial resolution feature maps)

%Pooling increases teh receptive field of the netwrok
%Modern image classification network models typically use a series of pooling operations to obtain multiscale context information. The pooling operation can effectively increase the receptive field of the neural
%network, which is beneficial for high-level computer vision tasks (such as classification). However, the
%pooling operation also has the problem of degrading the resolution of the picture: because the pooling
%operation is irreversible and will result in the loss of spatial information, which may result in the
%information of the small object cannot be reconstructed. This has a detrimental effect on the semantic
%segmentation task, because the semantic segmentation task requires both pixel-level precision and multiscale context inference.

An EDS network was trained to segment vessel locations because, although prior knowledge of the locations of vessels in the images was available for this \textit{in silico} study, this information will not always be available when imaging tissues \textit{in vivo}. Some technique for segmenting vessel positions from images is needed to enable the estimation of mean vessel sO$_2$ values from the output map. Therefore, a vessel segmentation network was trained to show that neural networks can be used to acquire accurate mean vessel sO$_2$ estimates without prior knowledge of vessels positions. As will be discussed in Section \ref{sec:Results}, the output of the segmentation network also provides some information about where estimates in the output sO$_2$ map may be more uncertain. This information can be used to improve mean vessel sO$_2$ estimates by disregarding values from these regions. Two separate networks were trained for each task to limit additional bias in the learned features that would arise from training a single network to learn both tasks simultaneously. In \cite{luke2019net}, two different loss functions were used in a single network trained to produce both an image of the vascular sO$_2$ distribution and an image of vessel locations. A different loss function was used for each task/branch of the network, where each function was arbitrarily assigned equal weights. Training two separate networks has the benefit that it removes the need to assign arbitrary weights to multiple loss functions that may be used to train a single network.

\begin{figure*}

  %\includegraphics[width=\textwidth]{2layer_UNET.png}
  \includegraphics[width=\textwidth]{EDS_arch.png}
  \caption{EDS network architecture. Blocks represent feature maps, where the number of feature maps generated by a convolutional layer is written above each block. Red arrows denote convolutional layers, green arrows denote maxpooling layers, dashed lines denote skip connections. }
  \label{fig:architecture}
\end{figure*}

 %e.g. lets say network detects presence of star in an image. Lets say a netowrk is trained so that a convolutional layer will output a feature map containing the location of all stars. If feature map is down pooled several times, it could become a single pixel feature map with high or low intensity depending on whether there is a star present anywhere the image. Information about where the stars are, is lost, but information about the the presence of the star is retained. Network only has to recognise that this pixel value has a high value in order to determine whether there are stars.

\subsection{Training Parameters}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Networks $A$ and $B$ were trained with 500 sets of images, corresponding to 500 different tissue models. An image of the true sO$_2$ distribution of the vessels was used as the ground truth for network $A$. A binary image of true vessel locations was used as the ground truth for network $B$. Network $A$ was trained for 98 epochs (loss curve shown in Fig. \ref{fig:Loss_curves}), while network $B$ was trained for 84. Both networks were trained with a batch size of 5 image sets, a learning rate of $10^{-4}$, and with Adam as the optimizer. Training was terminated with an early stopping approach, using a validation set of 5 examples. The networks were trained with the following error functionals $\epsilon(\theta_A)$, and $\epsilon(\theta_B)$ (the norm of the squared difference between the network outputs and the ground truth images):
\begin{equation}\label{error_function}
    \epsilon_{\theta_A} = ||A_{\theta_A}(p_{0}(x,\lambda)) - sO_{2}^{true}(x) ||^{2} _{2},
\end{equation}
and,
\begin{equation}\label{error_function}
    \epsilon_{\theta_B} = ||B_{\theta_B}(p_{0}(x,\lambda)) - seg^{true}(x) ||^{2} _{2},
\end{equation}
where $p_{0}(x,\lambda)$ are the multiwavelength images of each tissue model, $sO_{2}^{true}(x)$ and $seg^{true}(x)$ are the ground truth sO$_2$ and vessel segmentation images, and $\theta_A$ and $\theta_B$ are the network parameters. Once trained, the networks $A_{\theta_A}$, and $B_{\theta_B}$ were evaluated on 40 test examples.

\begin{figure}
\centering
  \includegraphics[width=.5\columnwidth]{Relative_loss_curves.png}

  \caption{Relative loss curves ($\epsilon_A$) for the sO$_2$ estimating network.}
  %Tissue models 20 and 28\
  \label{fig:Loss_curves}
\end{figure}

\subsection{Output Processing}
\label{sec:Image_processing}

The mean sO$_2$ of each vascular body was calculated using the voxels in each body that the segmentation network was confident contained vessels.

First, the indices associated with each major body in the segmentation network output ($V(x)$ where $x$ denotes the voxel index) were identified with the following method.

The output of the segmentation network $V(x)$ (where $x$ denotes the voxel index) was thresholded so all voxels with intensities $<0.2$ were set to zero, producing a new image $V'(x)$. This was done to remove small values which connected all the vessels into one large body. Then, the indices associated with each major body in $V'(x)$ were identified using the \verb bwlabeln() MATLAB function, generating a labelled image $L(x)$, where all the voxels belonging to each independent connected body were assigned the same integer value, and each body in the image was assigned a unique value to be identified by (e.g. all the voxels belonging to the largest body were assigned a value of one, all the voxels in the second largest body were assigned a value of two and so on).

Then, $V'(x)$ was thresholded so all voxels with intensities $<1$ were set to zero, producing a new image $V''(x)$. This was done to isolate voxels where the network was confident that vessels were present.

All the voxels in $L(x)$ that now had values of zero in $V''(x)$ were also set to zero, producing a new image $L_v(x)$. Bodies that were once a part of the same body before this thresholding step may now be in separate bodies, however, their voxel ID retains information about which body they originally belonged to. This allows for the mean sO$_2$ in each major vessel body to be calculated despite the thresholding (which removed voxels with low values in the segmentation network output) breaking up voxels which were once apart of the same body.

The mean sO$_2$ of the voxels sharing the same integer value in $L_v(x)$ were calculated using the corresponding values in the output of the sO$_2$ estimating network. The ground truth mean sO$_2$ of the voxels sharing the same integer value in $L_v(x)$ were calculated using the values from the ground truth sO$_2$ distribution.

%All the voxels in the output of the sO$_2$ estimating network $S(i)$ which had of zero in $V''(i)$ were also set to zero (producing a new image $S_v(i)$) to remove sO$_2$ predictions in voxels where the network was not confident that vessels were present.

%When performing the second thresholding step on the segmentation network output, voxels which were once part of the same vessel suddenly belong to two or more separate bodies. In order to avoid predicting the mean sO$_2$ in same vessel multiple times, the labelled image was generated before before the second thresholding step.

\section{Results and Discussion}
\label{sec:Results}

\begin{figure}

  \phantom{a}{\Large a.\par}

  \includegraphics[width=.5\columnwidth]{sO21.png}
  \includegraphics[width=.5\columnwidth]{Seg1.png}
  \phantom{a}{\Large b.\par}

  \includegraphics[width=.5\columnwidth]{sO22.png}
  \includegraphics[width=.5\columnwidth]{Seg2.png}
  \caption{2D slices of 3D network outputs and corresponding ground truth sO$_2$ and vessel segmentation images for two different tissue models (labelled a and b). }
  \label{fig:Graph_result}
\end{figure}
\begin{figure}

  \begin{tabular}{ll}
  %\phantom{a}{\Large a.\par}
  {\Large a.\par} & {\Large b.\par} \\
  \includegraphics[width=.4\columnwidth]{2_layer_UNet_plot.png}
  &
  %\phantom{a}{\Large b.\par}

  \includegraphics[width=.4\columnwidth]{plot_true_seg.png}
  \end{tabular}
  \caption{a.) Plot of the output mean vessel sO$_2$ vs the true values for all the vessels in 40 tissue models not used for training, calculated with the voxels belonging to each vessel as determined by the segmentation network output. b.) Plot of the mean sO$_2$ values for the same 40 tissue models calculated using the voxels known to belong to each vessel as determined by the ground truth vessel positions. These plots show that using the output of the segmentation network in combination with the output of the sO$_2$-estimating network significantly improves the accuracy of the estimates.}
  %Tissue models 20 and 28
  %20: 64, 65, 66
  %28: 60, 61, 62
  \label{fig:plot_result}

\end{figure}

The 3D image outputs of both the sO$_2$-estimating and segmentation networks were processed in order to calculate the mean sO$_2$ in each major vessel body using only the sO$_2$ estimates from voxels that the segmentation network was confident contained vessels (the reason for using the segmentation output was so that the mean vessel sO$_2$ could be calculated without \textit{a priori} knowledge of vessel locations that might not be available in an \textit{in vivo} scenario). More details about how this process was performed are provided in Section \ref{sec:Image_processing}. The mean of the absolute difference between the true mean vessel sO$_2$ and the output mean vessel sO$_2$ over all 40 sets of images was 4.4\%, and the standard deviation of the absolute difference between the true mean vessel sO$_2$ and the output mean vessel sO$_2$ was 4.5\% (some 2D image slices taken from the networks' 3D outputs are shown in Fig. \ref{fig:Graph_result}, and a plot of all the estimates is provided in Fig. \ref{fig:plot_result}.a). Therefore on average, the predicted mean vessel sO$_2$ was within 5\% of the true value. The mean difference between the true mean vessel sO$_2$ and the output mean vessel sO$_2$ was -0.3\% with an standard deviation of 6.3\%. The typical error for a mean vessel sO$_2$ estimate was thus between -6.6\% to 6.0\%.

To assess the effect that using the output of the segmentation network may have had on the accuracy of the sO$_2$ estimates, the mean sO$_2$ of each vascular body in the network output was estimated using the voxels known to belong to each vessel, as opposed to the voxels assigned to each body by the segmentation network output. Curiously, the accuracy of the estimates decreased when the ground truth vessel voxels were used for calculating mean sO$_2$ values. Fig. \ref{fig:plot_result}.b shows a plot of the results over 40 tissue models. The mean of the absolute value of the offset between the true value and the network output was 16.6\%, the mean offset was 16.2\%, and the standard deviation of the offset was 11.5\%. This suggests that regions where the segmentation network confidently classified as belonging to a vessel corresponded to regions where the sO$_2$ network was more accurate. Even though both networks are trained separately, they share the same input data, network architecture, and are trained with the same loss function where only the distribution of values in the corresponding ground-truth varies (continuous versus binary). As such it is not surprising that the learned mapping properties are similar and complement each other. The L2 loss was used for training network B (as opposed to a binary classification loss function that would normally be used for a segmentation task) to ensure that the network outputs would retain more information about the uncertainty of estimates.

%\textcolor{red}{Andreas, is the following explanation plausible?} Given that both networks have the same structure, and therefore the same limitations in what they can represent, and both are trained with the same training data - albeit with different ground truth images - the feature maps learned by both networks are likely to be similar. It is perhaps not surprising therefore that the learned sO$_2$ maps match the learned segmentation better than the true segmentation.

%Although a direct comparison between \cite{yang2019eda} and this study can not be made (as they did not report results for a wide range of different tissue models), this feature makes the technique more likely to produce accurate estimates in more complex tissue models, as it can use more complete information to generate estimates.

The accuracy of the output of both networks $A$ and $B$ decreased with the depth of the vessels, i.e. the distance from the detector array, as can be seen in Fig. \ref{fig:Graph_result}. There are a couple of reasons for why this might be the case. Firstly, image SNR decreases with depth as the amplitude of the signal decreases as fewer photons will propagate to/through the deeper tissue regions. Secondly, because the simulated sensor array has a limited-view planar geometry, it will detect less pressure data emitted from deeper within the tissue. The artefacts at greater depths are spread over more of space, introducing greater uncertainty as to the shape and location of the vessel. The output of both networks are least accurate in the deepest corners of each images, where the artefacts are the most significant.

Filters in a convolutional layer are the same wherever they are applied in the image - they are spatially invariant - and are therefore most suited to detecting features that are also spatially invariant. However, the limited-view artefacts are not; they are small close to the centre of the sensor array and become more significant further away. The multi-scale nature increases the receptive field and hence locality can be learned by the network. Nevertheless, we decided to limit the receptive field by using a slightly smaller network architecture than the classic U-Net, this way we retain uncertainty in the deeper tissue layers instead of introducing a learned bias.

%\textcolor{red}{[Andreas, we were hoping you might be able to provide further comment on why the output of both networks decreases with depth. In particular, how the spatial invariance of the convolutional filters might affect performance. Is the following reason plausible?]. Filters in a convolutional layer are the same wherever they are applied in the image - they are spatially invariant - and are therefore most suited to detecting features that are also spatially invariant. However, the limited-view artefacts are not; they are small close to the centre of the sensor array and become more significant further away. The multi-scale nature of the network can help ameliorate this problem, as features that at the top levels of the network span many voxels will span fewer at the lower levels, and so will become detectable by the convolutional filters.}

The 3D results shown in Figs. \ref{fig:Graph_result} and \ref{fig:plot_result}.a are of comparable accuracy to results from other groups obtained by training 2D convolutional neural networks to process 2D images (lacking the presence of reconstruction artefacts) of simpler 2D tissue models \cite{luke2019net,chen2020deep,yang2019quantitative}. The technique presented here was not only able to handle more complex tissue models (the tissue models presented here feature more realistic vascular architectures, and multiple skin layers with varying thicknesses and optical properties), but also took as the input data 3D images featuring noise and reconstruction artefacts. Unlike networks trained on 2D images sliced from 3D images of tissue models (such as those used in \cite{yang2019eda}), the 3D networks were able to use information from entire 3D image volumes to generate estimates. Because the fluence distribution and limited-view artefacts are 3D in nature, learning 3D features is more efficient than trying to learn to represent 2D sections/slices through 3D objects with 2D feature maps. This increases their ability to produce accurate estimates in more complex tissue models.%, where information about the absorbers and scatterers throughout the tissue volume may be required to formulate accurate estimates of the fluence and hence sO$_2$.

Despite being more sophisticated than other tissue models used to date, the tissue models used here were nevertheless created with some simplifying assumptions. Each skin layer was assigned a planar geometry, where the value of the optical properties associated with each layer at each wavelength remained constant within each layer (e.g. the scattering coefficient of the epidermis was constant within the epidermis layer). Although the absorption coefficient of each layer was varied for each tissue model, the scattering coefficient of each skin type remained constant (but did vary with wavelength). Other experimental factors that can affect image amplitude, such as the directivity of the acoustic sensors, were not incorporated into the simulation pipeline. It remains to be seen the extent to which these assumptions will hold true when this network is applied to \textit{in vivo} data. To ensure that networks initially trained on simplified simulated images can output accurate estimates when provided real images, networks may have to be modified with transfer training, taking advantage of datasets of real images \cite{pan2009survey,hauptmann2018model,wirkert2017physiological}. Looking beyond the complexity of the tissue models, there are other more fundamental challenges that will make the application to living tissue non-trivial. In order to train a network using a supervised learning approach with \textit{in vivo} data (or even to validate any technique for estimating sO$_2$ \textit{in vivo}), the corresponding ground truth sO$_2$ distribution must be available. It is unclear as to how this information might be acquired, and this poses a significant challenge that must be overcome to realize or validate the application of the technique. As an intermediate step towards generating \textit{in vivo} datasets, blood flow phantoms with tuneable sO$_2$ could be used to generate labelled data in conditions mimicking realistic imaging scenarios \cite{vogt2019photoacoustic,gehrung2019development}.
Although it is important to show that a network can cope with all the confounding effects present in real images of tissue, it is still interesting and important to know that the technique can cope with at least some of the challenges faced in such scenarios. This work provides an essential demonstration of the technique's ability to generate accurate 3D estimates from 3D image data despite the presence of some confounding experimental effects which distort image amplitude, and despite some variation in the distribution of tissue types and the distribution of vessels for each tissue model.

\section{Conclusions}
Data-driven approaches have been shown capable of recovering sample optical properties and maps of sO$_2$ from 2D PA images of fairly simple tissue models. However, because the fluence distribution  and limited-view artefacts are 3D, 2D networks are at a disadvantage as they must learn to represent 2D sections/slices through 3D objects with 2D feature maps. Networks that can process whole 3D images with 3D filters are more efficient as they can detect 3D features, and this increases their ability to produce accurate estimates in more complex tissue models. There may be cases where accurate sO$_2$ maps may only be generated with 3D network architectures. Therefore, to assess whether data-driven techniques have the potential to provide accurate estimates in realistic imaging scenarios, it is essential to demonstrate a neural network's ability to process 3D image data to generate sO$_2$ estimates. The capability of an EDS to generate accurate maps of vessel sO$_2$ and vessel locations from multiwavelength simulated images (containing noise and limited view artefacts) of tissue models featuring optically heterogeneous backgrounds (with varying absorption properties), and realistic vessel architectures was demonstrated. Regions where the segmentation output was confident in its predictions of vessel locations corresponded to more accurate regions in the sO$_2$-estimating network output. As a consequence, the accuracy of the network's mean vessel sO$_2$ estimates improved when the output of the segmentation network was used to determine vessel locations as opposed to the ground truth. In contrast to both analytical and iterative error-minimisation techniques, the networks were able to generate these estimates without total knowledge of each tissues' constituent chromophores, or an accurate image generation model - both of which would not normally be available in a typical \textit{in vivo} imaging scenario. This work shows that fully convolutional neural networks can process whole 3D images of tissues to generate accurate 3D images of vascular sO$_2$ distributions, and that accurate estimates can be generated despite some degree of variation in the distribution of tissue types, vessels, and the presence of noise and reconstruction artefacts in the data.

\appendix
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Use the below table for printout %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Noise Test}
\label{sec:Noise_test}
Noise was incorporated into the simulated images by adding it the the simulated pressure time series before the reconstruction step. Noise was added to each datapoint in the simulated pressure time series by adding a random number sampled from a Gaussian distribution with a standard deviation of 1\% of the maximum value over all time series data generated from the same image, resulting in realistic SNRs of 20.9 dB, 21.3 dB, 21.4 dB and 21.4 dB for a set of images of a single tissue model simulated at 784 nm, 796 nm, 808 nm and 820 nm, respectively. The details of this measurement are described in the following section.
\begin{enumerate}
    \item A single tissue model was defined.
    \item A fluence simulation was run 20 times (each run indexed with $r$) for each excitation wavelength ($\lambda$) with $10^{9}$ photons to produce $\Phi_r(x,\lambda)$, where $x$ indexes the voxels in the simulation output. The optical properties of the tissue model at each excitation wavelength were identical for all 20 runs.
    \item A set of initial pressure distributions, $I_r(x,\lambda)$, were generated from the fluence simulations:
    %
    \begin{equation}\label{eq:SNR_image_recon}
        I_r(x,\lambda) = \Phi_r(x,\lambda)\mu_a(x,\lambda)\Gamma.
    \end{equation}
    %
    \item The emission and detection of pressure time series were simulated in k-Wave to generate simulated pressure time series $p_{r,j}(t,\lambda)$, where $j$ indexes each time series produced by the simulation, and $t$ is the simulation time (simulation parameters were identical to those outlined in Section \ref{sec:Acoustic}).
    \item Some amount of noise $n_{r,j}(t,\lambda)$ was added to each point in each pressure time series:
    %
    \begin{equation}\label{eq:SNR_noise}
        \hat{p}_{r,j}(t,\lambda) = n_{r,j}(t,\lambda)+p_{r,j}(t,\lambda),
    \end{equation}
    %
    where $n_{r,j}(t,\lambda)$ was determined by sampling a random value from a Gaussian distribution with a standard deviation of $cn_{max}(\lambda)$, where $n_{max}(\lambda)$ is the max value of $p_{r,j}(t,\lambda)$ over all $j$ and $t$, for a given $\lambda$, and $r$ (i.e. the max value of all the time series for a given run at a given wavelength), while $c$ is the proportion of this value used to define the standard deviation.
    \item The images were reconstructed in k-Wave with time-reversal, to produce $I_r^{Recon}(x,\lambda)$.
    \item The mean and standard deviation for each voxel for each wavelength over all 20 runs was calculated with:
    %
    \begin{equation}\label{eq:SNR_mu_2}
        \mu(x,\lambda) = \frac{1}{20}\sum^{20}_{r=1}I_r^{Recon}(x,\lambda),
    \end{equation}
    %
    and
    %
    \begin{equation}\label{eq:SNR_sigma_2}
        \sigma(x,\lambda) = \left(\frac{1}{19}\sum^{20}_{r=1}(I_r^{Recon}(x,\lambda)-\mu(x,\lambda))\right)^{\frac{1}{2}}.
    \end{equation}
    %
     \item The SNR of each voxel at each wavelength was calculated with:
    %
    \begin{equation}\label{eq:pixel_SNR}
        SNR(x,\lambda) = 20\log_{10}(\frac{\mu(x,\lambda)}{\sigma(x,\lambda)}).
    \end{equation}
    %
    \item For each wavelength, the mean of the SNR values over all voxels $V$ was calculated with:
    %
    \begin{equation}\label{eq:SNR_mu_t}
        \mu_{SNR}(\lambda) = \sum^{V}_{x=1}SNR(x,\lambda).
    \end{equation}
    %

\end{enumerate}

Because the SNR depends on the optical properties of objects in the sample domain, the SNR will vary depending on the tissue model used for the test. Here, we only use a single tissue model with a single set of tissue properties to obtain some approximate idea of how much noise features in the simulated images.

\section{Optical Properties of Skin Layers}
\label{sec:Optical_properties}
The refractive index, anisotropy factor, optical absorption coefficient, and optical scattering coefficient of each tissue or chromophore are required to construct a tissue model for a MCXLAB simulation. Here, we tabulate expressions for computing the relevant quantities or list the values of certain quantities for various wavelengths in Table \ref{table:Skinproperties}. These values/resources were chosen as they featured data in the wavelength range for our simulations.

\clearpage

\begin{landscape}
\begin{table*}
\begin{longtable}[c]{llll}
\caption{Skin optical properties ($\lambda$ is given in nm).}
\label{table:Skinproperties}\\
\textbf{Tissue} & \textbf{Parameter} & \textbf{Value} & \textbf{Ref.} \\ \hline
\endhead
%
\hline
\endfoot
%
\endlastfoot
%
\multirow{6}{*}{Epidermis} & Optical absorption (cm\textsuperscript{-1}) & $\mu_{ae} = (C_M6.6(\lambda^{-3.33})(10^{11})) + (1-C_M)(0.244 + 85.3(\exp(-\frac{\lambda - 154}{66.2})))$ & \cite{StevenL.Jacques} \\
& Melanosome fraction \textit{C\textsubscript{M}} & 6\% for Caucasian skin, 40\% for pigmented skin & \cite{StevenJacques} \\
 & Reduced scattering (cm\textsuperscript{-1}) & $\mu_s' = 68.7(\frac{\lambda}{500})^{-1.16}$  & \cite{Jacques2013} \\
 & Refractive Index & 1.42 - 1.44 (700 nm - 900 nm) & \cite{Ding2006}  \\
 & Anisotropy & 0.95 - 0.8 (750 nm - 2500 nm)  & \cite{BASHKATOV2011} \\
 & Thickness & 0.1 mm & \\ \hline
\multirow{6}{*}{Dermis} & Optical absorption (cm\textsuperscript{-1}) & $\mu_{ad} = C_B\mu_{ab} + (1-C_B)(0.244 + 85.3(\exp(-\frac{\lambda - 154}{66.2})))$ & \cite{StevenJacques} \\
 & Blood volume fraction \textit{C\textsubscript{B}} & 0.2\% - 7\% & \cite{Yudovsky2011} \\
 & Reduced scattering (cm\textsuperscript{-1}) & $\mu_s' = 45.3(\frac{\lambda}{500})^{-1.292}$ & \cite{Jacques2013} \\
 & Refractive Index & $n = A + \frac{B}{\lambda^2} + \frac{C}{\lambda^4}$, where $A = 1.3696$, $B = 3.9168\textrm{x}10^3$, $C = 2.5588\textrm{x}10^3$ & \cite{Ding2006} \\
 & Anisotropy & 0.95 - 0.8 (750 nm - 2500 nm)  & \cite{BASHKATOV2011} \\
 & sO$_2$ & 40\% - 100\% & \cite{Yudovsky2011} \\ \hline
 \newpage
 Blood & Optical absorption (cm\textsuperscript{-1}) & $\mu_{ab} = C_{Hb}\alpha_{Hb} + C_{HbO_2}\alpha_{HbO_2}$ &\cite{StevenJacques} \\
 & Reduced scattering (cm\textsuperscript{-1}) & $22(\frac{\lambda}{500})^{-0.66}$ & \cite{Jacques2013} \\
 & Refractive Index & 1.36 (680 nm - 930 nm) & \cite{Lazareva2018} \\
 & Anisotropy & 0.994 (Roughly constant for variant wavelength and sO$_2$) & \cite{Bashkatov2005,Faber2004} \\ \hline
Hypodermis & Optical absorption (cm\textsuperscript{-1}) & 1.1 at 770 nm, 1.0 at 830 nm & \cite{Salomatina2006} \\
 & Reduced scattering (cm\textsuperscript{-1}) & 20.7 at 770 nm, 19.6 at 830nm & \cite{Salomatina2006} \\
 & Refractive index & 1.44 (680 nm - 930 nm) & \cite{Bashkatov2005} \\
 & Anisotropy & 0.8 (750 nm - 2500 nm) & \cite{valerytuchin2014} \\ \hline
\end{longtable}
\end{table*}

\end{landscape}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% \disclosures
\subsection*{Disclosures}
No conflicts of interest, financial or otherwise, are declared by the authors.

\subsection* {Acknowledgments}
The authors would like to thank Simon Arridge and Paul Beard for helpful discussions. The authors acknowledge support from the BBSRC London Interdisciplinary Doctoral Programme, LIDo, the European Union's Horizon 2020 research and innovation program H2020 ICT 2016-2017 under Grant agreement No. 732411, which is an initiative of the Photonics Public Private Partnership, the Academy of Finland Project 312123 (Finnish Centre of Excellence in Inverse Modelling and Imaging, 2018--2025) and the CMIC-EPSRC platform grant (EP/M020533/1).

%\subsection* {Data, Materials, and Code Availability}
%As relevant, the availability of data, materials, and/or software code used in the research results reported in the manuscript may be declared in this section. (Note: this section is required for the \textit{Journal of Biomedical Optics} and \textit{Neurophotonics}.) Provide specific access information or restrictions for data, materials, and computer code (i.e., links to repository access addresses with guidance on commercial or public access).

%%%%% References %%%%%

\bibliography{report}   % bibliography data in report.bib
\bibliographystyle{spiejour}   % makes bibtex use spiejour.bst

%%%%% Biographies of authors %%%%%

%\vspace{2ex}\noindent\textbf{Ciaran Bench} is...

%\vspace{2ex}\noindent\textbf{Andreas Hauptmann} is...

%\vspace{2ex}\noindent\textbf{Ben Cox} is...

%\vspace{1ex}
%\noindent Biographies and photographs of the other authors are not available.

\listoffigures
\listoftables

\end{spacing}
```


abstract: A diagnostic capable of recording spatially and temporally resolved X-ray self emission data was developed to characterise experiments on the MAGPIE pulsed-power generator. The diagnostic used two separate imaging systems: A pinhole imaging system with two dimensional spatial resolution and a slit imaging system with one dimensional spatial resolution. The two dimensional imaging system imaged light onto image plate. The one dimensional imaging system imaged light onto the same piece of image plate and a linear array of silicon photodiodes. This design allowed the cross-comparison of different images, allowing a picture of the spatial and temporal distribution of X-ray self emission to be established. The design was tested in a series of pulsed-power driven magnetic-reconnection experiments.
address: Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom; Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom; Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, 77 Massachusetts Avenue, Cambridge, MA 02139; Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom; Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom; Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom; Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom
author: Jack W. D. Halliday; Simon N. Bland; Jack. D. Hare; Susan Parker; Lee G. Suttle; Danny R. Russell; Sergey V. Lebedev
bibliography: Bibliography.bib
date: 2024-10-02
title: A Time-Resolved Imaging System for the Diagnosis of X-ray Self-Emission in High Energy Density Physics Experiments

# Introduction

The X-ray emission from high energy density (HED) plasmas is typically both transient and exhibits significant spatial variation. Traditionally, X-ray imaging data is captured either using gated X-ray detectors (for example micro-channel-plates ) or time-integrated detectors (such as image plate ). Time-gated detectors provide unambiguous information about the time interval in which X-ray emission occurs however they are expensive, bulky, introduce significant experimental complexity. Time-integrated detectors are compelling in the sense they are simple and inexpensive, however they provide no information about the time interval in which X-ray emission occurs and some dynamical aspects of experiments are also obscured by motion blurring.

The information gained from time-integrated X-ray detectors is often augmented with (time-resolved) detectors which convert X-ray flux into an electrical signal. Examples of such detectors include PCDs , vacuum X-ray diodes (XRDs), and silicon photodiodes. These are however typically fielded as spatially-integrated diagnostics.

In this paper, we describe a diagnostic which bridges the gap between the time-integrated and time-resolved X-ray diagnostics discussed above. This novel design made use of two separate X-ray imaging systems: A pinhole which imaged light onto image plate (time integrating), and a slit which imaged light onto both image plate and a linear array of silicon diodes (time-resolved). This meant that two-dimensional time-integrated, and one-dimensional time-resolved images were captured simultaneously. Features in the two-dimensional and one-dimensional images could be compared.

An approach similar to the one described here has previously been used to diagnose gas puff implosions on pulsed-power machines including Saturn and Double-EAGLE however time integrating X-ray detectors were not fielded alongside time-resolved ones.

There is an obvious analogy between the diagnostic we describe and an X-ray streak camera (which could also be set up to provide spatial resolution in one dimension). That said, much like MCP cameras, X-ray streaks are bulky, complex, and expensive diagnostics to field. They also have notable issues regarding both dynamic range and linearity of response. By contrast, the arrangement of silicon photodiodes we present is cheap, compact, and has a simpler response characteristic. That said, the diodes provide a coarser level of spatial resolution than an X-ray streak camera so should be seen as a complimentary technology which is best suited to harsh diagnostic environments, or situations in which a compact setup is required.

The design we describe could be applied to the diagnosis of a wide variety of HED plasmas, although it is best suited to systems which are either extended in one dimension or have a quasi one dimensional geometry. Examples of experiments which are extended in one dimension include gas-puff or wire-array Z-Pinch implosions. Examples of experiments with a quasi one dimensional geometry include the work on photoionisation fronts which currently form part of the work done by the Z Astrophysical Plasma Properties (ZAPP) collaboration .

The design we present was used to diagnose X-ray self emission in reconnection experiments performed on the MAGPIE generator at Imperial College . A diagnostic based on the design described here is also being developed to diagnose plasma dynamics in magnetic reconnection experiments performed on the Z-Machine at Sandia National Laboratories .

# Optical Design of the instrument

Figure shows a CAD model of the X-ray imaging system our diagnostic employed. The system used two separate apertures to image radiation from the experimental volume. The first of these was a pinhole which cast light (blue rays in figure ) onto a piece of image plate positioned on the same plane as the diode array. The second was a slit imaging system which cast light (red rays in figure ) onto both the image plate and the diode array. The slit imaging system provided one dimensional spatial resolution whilst the pinhole provided two dimensional resolution.

Diagrams showing the geometry of the X-ray apertures, and the baffling which was employed to prevent cross talk between the two imaging systems are shown in figure . Both of these components were laser-cut from 200 μm thick stainless steel, which was sufficiently opaque to block the relatively soft X-ray emission encountered in experiments performed on MAGPIE (nominal color temperature for emission is typically $\sim \SI{100}{\electronvolt}$). Inspection of laser cut components under a microscope revealed that features were accurate to $\sim \SI{20}{\micro\meter}$.

The projection of the ray traces onto the image plane is shown in figure . This diagram also shows the relative positions of the image plate and the diode array. The diode array was oriented so that it was spatially resolving in the same direction as the slit. The system was set up so that, for light cast from the object plane, the X-axis of the pinhole image could be directly related to the spatially resolved axis of the slit image. This enabled features seen in the signal from the diode array to be related to features seen in diagnostic images obtained in separate experiments.

# Details of X-ray detection technology

Time-integrated data was captured using Fuji-film image plate (BAS-TR). The image plates were scanned using a Fuji-Film FLA-5000 scanner.

The diode-array was an Opto Diode product with the model number AXUV20ELG. This consists of 20 photoconducting elements arranged linearly, as shown in figure . The size of each element is $0.75\times\SI{4.05}{\mm\squared}$ and that the centre-center separation between elements is 0.95 mm. This detector is silicon based and the thickness of the active layer is 50 μm. The spatial resolution was dominated by the geometric factors rather than diffractive effects. The width of the slit was chosen so the resolution was $\sim \SI{1}{\milli\meter}$ which is comparable to the separation between the photoconducting elements.

This diode array has a common cathode, and an individual anode for each photoconducting element. The cathode was biased to 9 V and the current through each element was relayed to a channel on an oscilloscope (HP Agilent 16500B with five 16532A digitizing oscilloscope cards) via a coaxial line terminated with a 50 Ω resistor. A passive low pass filter ($RC = \SI{10}{\kilo\ohm} \times \SI{220}{\nano\farad}$) was used on the (common) cathode in order to reduce the effects of electrical noise from the pulsed power generator and to maintain the bias voltage on the timescale of an experiment. No filtration was used between the anode side of the diodes and the channels of the oscilloscope.

The rise time of the detectors were empirically determined to be $\sim \SI{1}{\nano\second}$ by measuring their $V(t)$ response to radiation from a 200 ps pulsed laser (AXUV diodes have a low-level response in the visible). The discharge time for the capacitor is around $\SI{50}{\ohm} \times \SI{220}{\nano\farad} = \SI{11}{\micro\second}$ so the bias voltage was not significantly depleted on the timescale the an experiment (less than 0.5 μs).

The hydrodynamic timescale for typical experiments on MAGPIE is $\sim \SI{10}{\nano\second}$, so the temporal response of the detector is sufficient to resolve dynamics in our experiments. This temporal response is set (in part) by the bias voltage applied to the diodes and the bias voltage we used was selected to provide a sufficiently short temporal response without causing the diodes to breakdown.

We note that an alternative biasing scheme would have been to hold the cathode at ground, and have the bias voltage relayed to each individual photoconducting element, similar to the design reported in . This alternative biasing scheme would reduce the possibility of cross-talk between the diodes in the array.

As previously stated AXUV20ELG detectors use a 50 μm thick active layer of silicon. The detector exhibits a good level of quantum efficiency from $\sim \SI{100}{\electronvolt}$, up to $\sim \SI{10}{\kilo\electronvolt}$. For X-rays harder than this upper limit, the penetration of the (relatively thin) active layer limits the effective quantum efficiency.

# Application pulsed-power driven magnetic reconnection experiments

To validate the utility of the design described above, we fielded a prototype diagnostic in a series of pulsed-power driven magnetic reconnection experiments. Results from this magnetic reconnection platform have been previously reported in a number of publications . The setup was chosen here as the conditions of the plasmas produced are well characterised and the experimental morphology is extended in one dimension (rendering it suitable for diagnosis with the diode array).

The experimental setup is shown in figure . It consists of two inverse (or exploding) wire arrays which are driven in parallel by an intense current pulse. Each array acts as a radially diverging source of plasma carrying an azimuthal magnetic field. These plasma flows are continually driven for the duration of the generator's current pulse. For experiments on MAGPIE (1.4 MA, 500 ns current pulse), the typical conditions in the ablated plasma are $n_e \sim 10^{17} \; \si{\per\centi\meter\cubed}$; $T_e \sim \SI{15}{\electronvolt}$; $B_y \sim \SI{5}{\tesla}$.

As shown in the figure, the two plasma flows from the two arrays collide in a central interaction region, forming a reconnection layer. Where the two flows meet they carry oppositely directed magnetic fields. The magnetic fields reconnect, and magnetic energy is dissipated heating and accelerating the plasma flows. Within the central reconnection layer, the temperature and electron density of the plasma increase by around an order of magnitude. The spatial scale of the reconnection layer is typically $\SI{1}{\milli\meter} \times \SI{20}{\milli\meter}$.

Figure shows the regions of the experimental volume which were imaged to the diode array / image plate for the setup used in diagnostic tests. The figure shows that the slit was oriented to resolve structure across the reconnection layer (in the X-direction of the experiment). The magnification was $M=1$, so the width of the layer on the image plane was expected to be $\sim \SI{1}{\milli\meter}$.

Figure shows some example image plate data obtained using this experimental setup. For diagnostic tests, the pinhole was filtered with 40 μm polypropylene and the slit was filtered with 6.5 μm aluminium. The polypropylene filter transmits radiation from the both the plasma inflows and the reconnection layer whereas the aluminium transmits only the hardest radiation produced in the experiment. This means that the slit image only shows signal from the hottest plasma in the experiment, which is lies within the reconnection layer.

Figure shows the diode signals obtained in the same experiment. The X-axis of the plot is time (in nanoseconds) after current start; the signal from each diode is mapped to a position on the vertical axis which corresponds to its location in the array; and the colour-map indicates diode signal (in volts). As with the image plate data, the 6 μm aluminium filter applied to the slit absorbed the colder emission from the inflow region so we only expected to see a signal on the diodes which imaged the reconnection layer.

A comparison with image plate data enabled the data from the diode array to be related to features in the object plane. As expected, only the diode which imaged the reconnection layer (positioned at $X=\SI{0}{\milli\meter}$) shows a significant signal. This response persists during the interval 250-500 ns after current start. The presence of multiple peaks in the data may be related to the disruption of the reconnection layer at late times. The amplitude of the peaks are generally as expected from other diagnostics, however we currently lack the detailed knowledge of the emission spectrum from the layer which is required to make this statement more quantitative.

For the purpose of this paper, the key points to take away from this section is that the emission from the layer is imaged onto the diode array and that comparing data from the array with images captured using a time integrated detector allows the signal on a particular diode to be related to regions in the experimental volume, without relying on precise alignment of the diagnostic with respect to the experimental load.

# Conclusions

We presented a design for a time-resolved X-ray imaging system to characterise X-ray self emission in HED experiments. The diagnostic incorporates two independent imaging systems, one with two-dimensional, and one with one-dimensional spatial resolution. Both imaging systems are used to record time-integrated X-ray self-emission images onto image plate. The one-dimensional imaging system was also used in combination with a linear array of 20 photodiodes to record time-resolved data. A comparison between the image-plate data from the two imaging systems enabled the features in one-dimensional images to be understood in the context of an experimental morphology which varied in two dimensions. This diagnostic is applicable to HED experiments which are either extended in one dimension or have a quasi one dimensional geometry.

# Acknowledgements

This work was supported by the U.S. Department of Energy (DOE) under Award Nos. DE-SC0020434, DE-NA0003764, DE-F03-02NA00057, and DE-SC-0001063; and by the U.S. Defence Threat Reduction Agency (DTRA) under award number HDTRA1-20-1-0001.

# Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.


abstract: This paper addresses the problem of continuous gesture recognition from sequences of depth maps using Convolutional Neural networks (ConvNets). The proposed method first segments individual gestures from a depth sequence based on quantity of movement (QOM). For each segmented gesture, an Improved Depth Motion Map (IDMM), which converts the depth sequence into one image, is constructed and fed to a ConvNet for recognition. The IDMM effectively encodes both spatial and temporal information and allows the fine-tuning with existing ConvNet models for classification without introducing millions of parameters to learn. The proposed method is evaluated on the Large-scale Continuous Gesture Recognition of the ChaLearn Looking at People (LAP) challenge 2016. It achieved the performance of 0.2655 (Mean Jaccard Index) and ranked $3^{rd}$ place in this challenge.
author: Pichao Wang, Wanqing Li, Song Liu, Yuyao Zhang, Zhimin Gao and Philip Ogunbona

Advanced Multimedia Research Lab, University of Wollongong, Australia

firstname.lastname@example.com, {wanqing, email@example.com,
{yz606, firstname.lastname@example.com}, email@example.com
title: Large-scale Continuous Gesture Recognition Using
  Convolutional Neural Networks

# Introduction

Gesture and human action recognition from visual information is an active research topic in Computer Vision and Machine Learning. It has many potential applications including human-computer interactions and robotics. Since first work  on action recognition from depth data captured by commodity depth sensors (e.g., Kinect) in 2010, many methods  for action recognition have been proposed based on specific hand-crafted feature descriptors extracted from depth or skeleton data. With the recent development of deep learning, a few methods have also been developed based on Convolutional Neural Networks (ConvNets)  and Recurrent Neural Networks (RNNs) . However, most of the works on gesture/action recognition reported to date focus on the classification of individual gestures or actions by assuming that instances of individual gestures and actions have been isolated or segmented from a video or a stream of depth maps/skeletons before the classification. In the cases of continuous recognition, the input stream usually contains unknown numbers, unknown orders and unknown boundaries of gestures or actions and both segmentation and recognition have to be solved at the same time.

There are three common approaches to continuous recognition. The first approach is to tackle the segmentation and classification of the gestures or actions separately and sequentially. The key advantages of this approach are that different features can be used for segmentation and classification and existing classification methods can be leveraged. The disadvantages are that both segmentation and classification could be the bottleneck of the systems and they can not be optimized together. The second approach is to apply classification to a sliding temporal window and aggregate the window based classification to achieve final segmentation and classification. One of the key challenges in this approach is the difficulty of setting the size of sliding window because durations of different gestures or actions can vary significantly. The third approach is to perform the segmentation and classification simultaneously.

This paper adopts the first approach and focuses on robust classification using ConvNets that are insensitive to inaccurate segmentation of gestures. Specifically, individual gestures are first segmented based on quantity of movement (QOM)  from a stream of depth maps. For each segmented gesture, an Improved Depth Motion Map (IDMM) is constructed from its sequence of depth maps. ConvNets are used to learn the dynamic features from the IDMM for effective classification. Fig.  shows the framework of the proposed method.

The rest of this paper is organized as follows. Section II briefly reviews the related works on video segmentation and gesture/action recognition based on depth and deep learning. Details of the proposed method are presented in Section III. Experimental results on the dataset provided by the challenge are reported in Section IV, and Section V concludes the paper.

# Related Work

## Video Segmentation

There are many methods proposed for segmenting individual gestures from a video. The popular and widely used method employs dynamic time warping (DTW) to decide the delimitation frames of individual gestures . Difference images are first obtained by subtracting two consecutive grey-scale images and each difference image is partitioned into a grid of $3\times3$ cells. Each cell is then represented by the average value of pixels within this cell. The matrix of the cells in a difference image is flattened as a vector called motion feature and calculated for each frame in the video excluding the final frame. This results in a $9\times{}(K-1)$ matrix of motion features for a video with $K$ frames. The motion feature matrix is extracted from both test video and training video which consists of multiple gestures. The two matrices are treated as two temporal sequences with each motion feature as a feature vector at an instant of time. The distance between two feature vectors is defined as the negative Euclidean distance and a matrix containing DTW distances (measuring similarity between two temporal sequences) between the two sequences is then calculated and analysed by Viterbi algorithm  to segment the gestures.

Another category of gesture segmentation methods from a multi-gesture video is based on appearance. Upon the general assumption that the *start* and *end* frames of adjacent gestures are similar, correlation coefficients  and K-nearest neighbour algorithm with histogram of oriented gradient (HOG)  are used to identify the *start* and *end* frames of gestures. Jiang et al.  proposed a method based on quantity of movement (QOM) by assuming the same *start* pose among different gestures. Candidate delimitation frames are chosen based on the global QOM. After a refining stage which employs a sliding window to keep the frame with minimum QOM in each windowing session, the *start* and *end* frames are assumed to be the remained frames. This paper adopts the QOM based method and its details will be presented in Section .

## Depth Based Action Recognition

With Microsoft Kinect Sensors researchers have developed methods for depth map-based action recognition. Li et al. sampled points from a depth map to obtain a bag of 3D points to encode spatial information and employ an expandable graphical model to encode temporal information . Yang et al. stacked differences between projected depth maps as a depth motion map (DMM) and then used HOG to extract relevant features from the DMM. This method transforms the problem of action recognition from spatio-temporal space to spatial space. In , a feature called Histogram of Oriented 4D Normals (HON4D) was proposed; surface normal is extended to 4D space and quantized by regular polychorons. Following this method, Yang and Tian cluster hypersurface normals and form the polynormal which can be used to jointly capture the local motion and geometry information. Super Normal Vector (SNV) is generated by aggregating the low-level polynormals. In , a fast binary range-sample feature was proposed based on a test statistic by carefully designing the sampling scheme to exclude most pixels that fall into the background and to incorporate spatio-temporal cues.

## Deep Leaning Based Recogntiion

Existing deep learning approach can be generally divided into four categories based on how the video is represented and fed to a deep neural network. The first category views a video either as a set of still images  or as a short and smooth transition between similar frames , each color channel of the images is fed to one channel of a ConvNet. Although obviously suboptimal, considering the video as a bag of static frames performs reasonably well. The second category represents a video as a volume and extends ConvNets to a third, temporal dimension  replacing 2D filters with 3D equivalents. So far, this approach has produced little benefits, probably due to the lack of annotated training data. The third category treats a video as a sequence of images and feeds the sequence to an RNN . An RNN is typically considered as memory cells, which are sensitive to both short as well as long term patterns. It parses the video frames sequentially and encodes the frame-level information in their memory. However, using RNNs did not give an improvement over temporal pooling of convolutional features  or over hand-crafted features. The last category represents a video as one or multiple compact images and adopts available trained ConvNet architectures for fine-tuning . This category has achieved state-of-the-art results of action recognition on many RGB and depth/skeleton datasets. The proposed gesture classification in this paper falls into the last category.

# Proposed Method

The proposed method consists of two major components, as illustrated in Fig. : video segmentation and construction of Improved Depth Motion Map (IDMM) from a sequence of depth maps as the input to ConvNets. Given a sequence of depth maps consisting of multiple gestures, the *start* and *end* frames of each gesture are identified based on quantity of movement (QOM) . Then, one IDMM is constructed by accumulating the absolute depth difference between current frame and the *start* frame for each gesture segment. The IDMM goes through a pseudo-color coding process to become a pseudo-RGB image as an input to a ConvNet for classification. The main objective of pseudo-color coding is to enchance the motin pattern captured by the IDMM. In the rest of this section, video segmentation, construction of IDMM, pseudo-color coding of IDMM, and training of the ConvNets are explained in detail.

```latex
\begin{table*}[!ht]\centering
\caption{Information of the ChaLearn LAP ConGD Dataset. \label{table1}}
\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
Sets &\# of labels &\# of gestures & \# of RGB videos & \# of depth videos & \# of subjects & label provided & temporal segment provided\\
\hline
Training & 249 & 30442 & 14134 & 14134 & 17 & Yes & Yes\\
\hline
Validation & 249 & 8889 & 4179 & 4179 & 2 & No & No\\
\hline
Testing & 249 &  8602 & 4042 & 4042 & 2 & No & No\\
\hline
All & 249 & 47933 & 22535 & 22535 & 21 & - & -\\
\hline
\end{tabular}
\end{table*}
```

## Video Segmentation

Given a sequence of depth maps that contains multiple gestures, The *start* and *end* frames of each gesture is detected based on quantity of movement (QOM)  by assuming that all gestures starts from a similar pose, referred to as Neural pose. QOM between two frames is a binary image obtained by applying $f(\cdot)$ pixel-wisely on the difference image of two depth maps. $f(\cdot)$ is a step function from 0 to 1 at the ad hoc threshold of 60. The global QOM of a frame at time $t$ is defined as QOM between Frame $t$ and the very first frame of the whole video sequence. A set of frame indices of candidate delimitation frames was initialised by choosing frames with lower global QOMs than a threshold. The threshold was calculated by adding the mean to twice the standard deviation of global QOMs extracted from first and last $12.5\%$ of the averaged gesture sequence length $L$ which was calculated from the training gestures. A sliding window with a size of $\frac{L}{2}$ was then used to refine the candidate set and in each windowing session only the index of frame with a minimum global QOM is retained. After the refinement, the remaining frames are expected to be the deliminating frames of gestures.

## Construction Of IDMM

Unlike the Depth Motion Map (DMM) which is calculated by accumulating the thresholded difference between consecutive frames, two extensions are proposed to construct an IDMM. First, the motion energy is calculated by accumulating the absolute difference between the current frame and the Neural pose frame. This would better measure the slow motion than original DMM. Second, to preserve subtle motion information, the motion energy is calculated without thresholding. Calculation of IDMM can be expressed as:

$$\begin{aligned}
IDMM = \sum_{i=1}^{N}|(\text{depth of frame})^{i} -
\text{depth of Neural frame}|
\end{aligned}$$ where $i$ denotes the index of the frame and $N$ represents the total number of frames in the segmented gesture. For simplicity, the first frame of each segment is considered as the Neural frame.

## Pseudocoloring

In their work Abidi et al.  used color-coding to harness the perceptual capabilities of the human visual system and extracted more information from gray images. The detailed texture patterns in the image are significantly enhanced. Using this as a motivation, it is proposed in this paper to code an IDMM into a pseudo-color image and effectively exploit/enhance the texture in the IDMM that corresponds to the motion patterns of actions. In this work, a power rainbow transform is adopted. For a given intensity $I$, the power rainbow transform encodes it into a normalized color $(R^{*},G^{*},B^{*})$ as follows: $$\label{eq7}
  \begin{split}
    &R^{*} = \{(1 + cos(\frac{4\pi}{3*255}I))/2\}^{2}\\
    &G^{*} = \{(1 + cos(\frac{4\pi}{3*255}I-\frac{2\pi}{3}))/2\}^{2}\\
    &B^{*} = \{(1 + cos(\frac{4\pi}{3*255}I-\frac{4\pi}{3}))/2\}^{2},
  \end{split}$$

where $R^{*}$, $G^{*}$ and $B^{*}$ are the normalized RGB values through the power rainbow transform. To code an IDMM, linear mapping is used to convert IDMM values to $I\in[0,255]$ across per image.

The resulting IDMMs are illustrated as in Fig. .

## Network Training & Classification

One ConvNet is trained on the pseudo-color coded IDMM. The layer configuration of the ConvNets follows that in . The ConvNet contains eight layers, the first five are convolutional layers and the remaining three are fully-connected layers. The implementation is derived from the publicly available Caffe toolbox based on one NVIDIA Tesla K40 GPU card.

The training procedure is similar to that in . The network weights are learned using the mini-batch stochastic gradient descent with the momentum set to 0.9 and weight decay set to 0.0005. All hidden weight layers use the rectification (RELU) activation function. At each iteration, a mini-batch of 256 samples is constructed by sampling 256 shuffled training color-coded IDMMs. All color-coded IDMMs are resized to $256 \times
256$. The learning rate for fine-tuning is set to $10^{-3}$ with pre-trained models on ILSVRC-2012, and then it is decreased according to a fixed schedule, which is kept the same for all training sets. For the ConvNet the training undergoes 20K iterations and the learning rate decreases every 5K iterations. For all experiments, the dropout regularisation ratio was set to 0.5 in order to reduce complex co-adaptations of neurons in nets.

Given a test depth sequence, a pseudo-colored IDMM is constructed for each segmented gesture and the trained ConvNet is used to predict the gesture label of the segment.

# Experiments

In this section, the Large-scale Continuous Gesture Recognition Dataset of the ChaLearn LAP challenge 2016 (ChaLearn LAP ConGD Dataset))  and evaluation protocol are described. The experimental results of the proposed method on this dataset are reported and compared with the baselines recommended by the chellenge organisers.

## Dataset

The ChaLearn LAP ConGD Dataset is derived from the ChaLearn Gesture Dataset (CGD) . It has 47933 RGB-D gesture instances in 22535 RGB-D gesture videos. Each RGB-D video may contain one or more gestures. There are 249 gestures performed by 21 different individuals. The detailed information of this dataset is shown in Table . In this paper, only depth data was used in the proposed method. Some samples of depth maps are shown in Fig. .

## Evaluation Protocol

The dataset was divided into training, validation and test sets by the challenge organizers. All three sets include data from different subjects and the gestures of one subject in validation and test sets do not appear in the training set.

Jaccard index (the higher the better) is adopted to measure the performance. The Jaccard index measures the average relative overlap between true and predicted sequences of frames for a given gesture. For a sequence $s$, let $G_{s,i}$ and $P_{s,i}$ be binary indicator vectors for which 1-values correspond to frames in which the $i^{th}$ gesture label is being performed. The Jaccard Index for the $i^{th}$ class is defined for the sequence $s$ as:

$$J_{s,i} = \dfrac{G_{s,i}\bigcap P_{s,i}}{G_{s,i}\bigcup P_{s,i}},$$ where $G_{s,i}$ is the ground truth of the $i^{th}$ gesture label in sequence $s$, and $P_{s,i}$ is the prediction for the $i^{th}$ label in sequence $s$.

When $G_{s,i}$ and $P_{s,i}$ are empty, $J_{(s,i)}$ is defined to be 0. Then for the sequence $s$ with $l_{s}$ true labels, the Jaccard Index $J_{s}$ is calculated as:

$$J_{s} = \dfrac{1}{l_{s}}\sum_{i=1}^{L}J_{s,i}.$$ For all testing sequences $S = {s_{1},...,s_{n}}$ with $n$ gestures, the mean Jaccard Index $\overline{J_{S}}$ is used as the evaluation criteria and calculated as: $$\overline{J_{S}} = \dfrac{1}{n}\sum_{j=1}^{n}J_{s_{j}}.$$

## Experimental Results

The results of the proposed method on the validation and test sets and their comparisons to the results of the baseline methods  (MFSK and MFSK+DeepID) are shown in Table . The codes and models can be downloaded at the author's homepage <https://sites.google.com/site/pichaossites/>
[.](https://sites.google.com/site/pichaossites/)

|     Method      |    Set     | Mean Jaccard Index $\overline{J_{S}}$ |
|:---------------:|:----------:|:-------------------------------------:|
|      MFSK       | Validation |                0.0918                 |
|   MFSK+DeepID   | Validation |                0.0902                 |
| Proposed Method | Validation |              **0.2403**               |
|      MFSK       |  Testing   |                0.1464                 |
|   MFSK+DeepID   |  Testing   |                0.1435                 |
|    Proposed     |  Testing   |              **0.2655**               |

Accuracies of the proposed method and baseline methods on the ChaLearn LAP ConGD Dataset.

The results showed that the proposed method significantly outperformed the baseline methods, even though only single modality, i.e. depth data, was used while the baseline method used both RGB and depth videos.

The first three winners' results are summarized in Table . We can see that our method is among the top performers and our recognition rate is very close to the best performance of this challenge (0.265506 vs. 0.269235&0.286915), even though we only used depth data for proposed method. Regarding computational cost, our implementation is based on CUDA 7.5 and Matlab 2015b, and it takes about 0.8s to process one depth sequence for testing in our workstation equipped with 8 cores CPU, 64G RAM, and Tesla K40 GPU.

| Rank |      Team       | Mean Jaccard Index $\overline{J_{S}}$ |
|:----:|:---------------:|:-------------------------------------:|
|  1   |    ICT_NHCI     |               0.286915                |
|  2   |     TARDIS      |               0.269235                |
|  3   | **AMRL (ours)** |               0.265506                |

Comparsion the performances of the first three winners in this challenge. Our team ranks the third place in the ICPR ChaLearn LAP challenge 2016.

# Conclusions

This paper presents an effective yet simple method for continuous gesture recognition using only depth map sequences. Depth sequences are first segmented so that each segmentation contains only one gesture and a ConvNet is used for feature extraction and classification. The proposed construction of IDMM enables the use of available pre-trained models for fine-tuning without learning afresh. Experimental results on ChaLearn LAP ConGD Dataset verified the effectiveness of the proposed method. How to exactly extract the neutral pose and fuse different modalities to improve the accuracy will be our future work.

# Acknowledgment

The authors would like to thank NVIDIA Corporation for the donation of a Tesla K40 GPU card used in this challenge.