Muhammad Ishfaq Hussain,Muhammad Aasim Rafique,Wan-Gil Jung,Bong-Joong Kim,Moongu Jeon
Segmentation and Morphology Computation of a Spiky Nanoparticle Using the Hourglass Neural Network
Morphological measurements of nanoparticles in electron microscopy images are tedious, laborious, and often succumb to human errors. Deep learning methods in artificial intelligence (AI) paved the way for automated image understanding. This work proposes a deep neural network (DNN) for the automated segmentation of a Au spiky nanoparticle (SNP) in electron microscopic images, and the network is trained with a spike-focused loss function. The segmented images are used for the growth measurement of the Au SNP. The auxiliary loss function captures the spikes of the nanoparticle, which prioritizes the detection of spikes in the border regions. The growth of the particles measured by the proposed DNN is as good as the measurement in manually segmented images of the particles. The proposed DNN composition with the training methodology meticulously segments the particle and consequently provides accurate morphological analysis. Furthermore, the proposed network is tested on an embedded system for integration with the microscope hardware for real-time morphological analysis.
Segmentation and Morphology Computation of a Spiky Nanoparticle Using the Hourglass Neural Network

Morphological measurements of nanoparticles in electron microscopy images are tedious, laborious, and often succumb to human errors. Deep learning methods in artificial intelligence (AI) paved the way for automated image understanding. This work proposes a deep neural network (DNN) for the automated segmentation of a Au spiky nanoparticle (SNP) in electron microscopic images, and the network is trained with a spike-focused loss function. The segmented images are used for the growth measurement of the Au SNP. The auxiliary loss function captures the spikes of the nanoparticle, which prioritizes the detection of spikes in the border regions. The growth of the particles measured by the proposed DNN is as good as the measurement in manually segmented images of the particles. The proposed DNN composition with the training methodology meticulously segments the particle and consequently provides accurate morphological analysis. Furthermore, the proposed network is tested on an embedded system for integration with the microscope hardware for real-time morphological analysis.

Nanoparticles are studied for their various uses controlled by their morphological properties. The change in morphological properties and their accurate observation unlocks the application of nanoparticles in the fields of target drug delivery, surface plasmon resonance (SPR) sensing, surface-enhanced Raman scattering (SERS) detection, biosensing, and catalysis. Au spiky nanoparticles (SNPs) are studied in electron micropscopic images, and various applications are discussed with the importance of the morphological analysis of the nanoparticle. However, determining the peculiar shape of the SNP during a period of its growth is a strenuous and laborious exercise. Automation of laborious jobs is a developing norm and taking its course from established industries to research and development. Along with many computational fields, the applied natural sciences are benefiting from recent deep learning trends in artificial intelligence (AI). Until now, the automated analysis of cell morphology has been performed using conventional techniques in computer vision and machine learning. A support vector machine with handcrafted features was used by Lohrer et al. to analyze dendritic cell maturation. where electron microscopic images (AFM and SEM) of dendritic cells are acquired and processed using a conventional morphological operation. Microscopic images are processed with noise removal, and image features are computed using histogram of oriented gradients (HOG) and classified using the error correcting output codes (ECOC) technique. A recent work evaluated multiple state-of-the-art deep learning models for the segmentation of nanoparticles in bright-field transmission electron microscopy (BF-TEM) and environmental transmission electron microscopy (E-TEM) images. The BF-TEM images are for nanocatalyst Pt nanoparticles, whereas the E-TEM images are for nanoparticles, and both particles in the tested images are regularly shaped. This study considers the automated morphology analysis of irregular-shaped SNPs over the course of their growth during an experiment. Conventional unsupervised learning techniques are used to monitor the growth of the SNP. However, deep learning techniques cannot detect finer boundaries of the particle. The analysis needs a fine extraction of the particle and separate it from the environment. This study uses a state-of-the-art UNet deep learning network for segmentation. The peculiar morphology of the SNP challenged the state-of-the-art capacity and did not extract the finer spikes of the particle. Although the dice coefficient loss function improves border detection, the sharp spikes of SNP are not accurately segmented. In this study, an additional term is proposed in the loss function to improve the sharp spikes and boundary segmentation of the particle. The implementation of the proposed loss term is easy with the change in the cross-entropy loss function. The following are the main contributions tested in this study: Automated morphology analysis of a Au SNP. A boundary loss to favor learning of the irregular shape spikes in segmentation. Implementation of the proposed loss in UNet and testing on an embedded system. The study is structured as follows: The proposed methodology and data set preparation are described in the Methodology section. The Experimentation and Results section outlines the necessary experimental setup for the study and provides a detailed analysis of results, as well as a comparison with other methods. In the end, the Conclusion section concludes this study.

The morphology of a Au SNP is automated using semantic segmentation of the particle in TEM images. Deep neural networks (DNNs) are state-of-the-art in-object segmentation in images. There are two pertinent challenges using SOTA with the Au SNP data set. First, there are few images available with visible spikes. Second, the particle overlaps with noise on the border. The two challenges are solved by exploring the hard inductive bias of DNN and identifying a border favoring loss function for training.

This study uses a variant of UNet, a DNN with a loose encoder-decoder architecture. The encoder is a sequence of convolutional layers, and the decoder is a sequence of deconvolution layers. The proposed network uses the same architecture with different compositions of the layers in the network, and Figure 1 shows a detailed composition of the layers. The image Ii, a H × W × C, is clamped to the encoder, which down-samples the image and extracts the features. The encoder is divided into modules (Figure 1 enM-i) of the same spatial resolution, and within each module the resolution of the receptive field is increased by twice from each layer to the next layer. The layers in the encoder are convolution layers and use a kernel of size 3 × 3 for all convolutions. The decoder starts with the output of as the input and the upsamples to generate the spatial resolution of H × W of the segmented image size. The decoder is also divided into modules (Figure 1 deM-i) of the same spatial resolution within the module. The spatial resolution increases from one module to the next in the decoder. The receptive field resolution is increased with the input from an adjacent encoder layer like , where “↷” is a concatenation operation applied with the same spatial resolution. The receptive fields decrease by a factor of 2 with increasing q in the decoder and are exactly equal to the same number of filters in the corresponding encoder module. is passed to a final layer that classifies the pixel location, i.e., whether it is a particle pixel or a background pixel.

DL networks are composed of activation functions and the connection between the activation functions. The connection is the parameters, θ, of a network that are tuned with learning algorithms. A gradient descent technique updates the parameters learning the deviation of the network predictions, , from target values, yi, for an input image Ii. The deviation is measured using a loss function. A commonly used loss function for classification is cross-entropy loss, , for each pixel in the image, where C is the number of classes and mn is the location of the pixels. Cross-entropy loss gives equal weight to all locations in the images; however, a challenge with images in the current data set is that the border of the particle in images contains sharp spikes. A dice coefficient is used to consider the impact of the scarcity of boundary data values compared to nonboundary values. The loss function with a dice coefficient captures a global impact of the prediction and target values. A peculiarity of the Au SNP data set gives prior knowledge about the boundary of the particle that helps us to design a loss function term additional to the dice coefficient favoring the prediction at the boundary of the particle. The additional loss term enforces the additional penalty for loss in the boundary pixel, b. The loss function in this study uses a combined loss with the cross-entropy, the dice coefficient, and the additional boundary-favoring term.Here α, β, and γ are weights to adjust the contributions in the loss for each term in the loss. p is the probability of the particle pixel, and q is the probability of other pixels. C is the number of classes. The additional loss is easily achieved in the implementation with another cross-entropy loss with the edges mask generated as the target.

The primary objective of this study is the analysis and prediction of the morphology of a Au SNP. There are two important components of the nanoparticle, the main body and the spikes. The study proposes automated methods to generate statistics for the size of a complete nanoparticle and the number of visible spikes. Moreover, it generates statistics about the height and sharpness of the largest spike. The size of a nanoparticle is computed first, followed by the automated calculation of the aforementioned spike-related parameters. It should be noted that the number of spikes visible in the 2D image of the Au SNP may vary from the number of spikes present in the actual 3D object.

Segmentation of the particle from TEM images over the course of its growth significantly reduces the challenge of computing the size statistics. A contour enclosing the segmented particle is drawn using the border-following technique. Properties of circular regions give a fine measurement of the morphology of the particle in pixels, and the region is drawn enclosing the contours around the segments. The following formulations are used to calculate the actual size of the particle from the pixel and circle statistics:where r is the radius of the circle. The formulation computes the automated statistics of the morphology of a particle and its size over the course of its growth during experiments.

The Au SNP has peculiar shapes with spikes on the body surface that are visible in the 2D image. Spikes grow over the period of growth of the particle, and the morphology of the spikes is automatically analyzed after postprocessing of the segmented nanoparticle. The number of spikes is calculated by detaching the spikes from the body of the particle using image morphological techniques in computer vision. In this study, an elliptical kernel is used to detach the spikes from the body, and Figure 7 shows the segmented results of detached spikes. Later, a connected component analysis is performed to label and count the number of spikes. Additionally, the size of the largest spike is calculated by measuring its height using the Hough circle enclosing the spike. Another important fact is the sharpness of the spike, which can be calculated using a corner detection technique and is left for exploration in future studies.

There are two stages of experimental setup that are pertinent to this study. The first stage is a setup to grow the particle in a solution and to capture electron microscopic data. The second stage is the use of digital image data to design an algorithm that automatically generates morphological statistics of the particle. The experimental setup of the computational system is further designed with two compositions. The initial experimentation is performed on a PC with an Intel Corei7 processor, 16 GB of RAM, and an Nvidia GeForce RTX 3090 GPU. Another particular setup is created with an Nvidia Jetson Nano embedded board. The latter setup is a low-cost solution to integrate the automated morphology operations with the electron microscopic hardware.

The hydrogen tetrachloroaurate (III) (HAuCl4·3H2O) solution and deionized (DI) water (Milli-Q) were purchased from Alfar Asear and Fisher Chemical, respectively. A stock solution of HAuCl4 was diluted with DI water to form solutions of 20 mM. Due to their light-sensitive nature, the solutions were stored in an opaque glass bottles and covered with aluminum foil.

To observe the process of SNP growth induced by electron beam radiolysis, we used a liquid cell transmission electron microscope (LCTEM) holder purchased from Hummingbird Scientific. An LCTEM holder includes an inlet and an outlet liquid path lines, both of which are built for the continuous flow of the solution into the microsize aquarium. The aquarium is fabricated by an assembly of two (top and bottom) chips of silicon nitride (Si3N4) membranes whose dimensions are 50 × 200 μm2, and the liquid thickness can be manipulated by a choice of spacers that are inserted between the two chips. Prior to assembling the chips, we cleaned the two chips with ethanol and deionized water and then dried them in a desiccator. The Si3N4 membrane was turned hydrophilic by plasma cleaning for 30 s. Subsequently, the chips were assembled in the holder, then they were inserted into the vacuum station (1.6 × 10–6 Torr) for several hours while the deionized water flowed at 10 μL/min to confirm the vacuum safety within the TEM (Tecnai 300 keV) and to remove bubbles in the solution. Finally, the holder was loaded into the TEM to perform the experiments. The images were taken using the digital micrograph-3 program.

In this work, the electron beam is used to grow Au particles from a solution of Au ions by radiolysis in which the electron beam forms hydrated electron (), among other species. To adjust the dose rate of the transmitted electron beam, which is an important factor for Au particle formation, we changed spot sizes and gun lenses. The dose rate of the transmitted electron beam was measured using a viewing screen. Electron beam sizes and dose rates used for our experiments were in the ranges of 1.84 μm and 57–360 e–/nm2·s, respectively. A pH value of below 2 is maintained for HAuCl4. The time for each experiment is less than 1 min. The conditions mentioned above also helped maintain no bubble condition in the cell. The experiments are performed near the edgees of the membranes that had constant liquid thickness induced by the difference in pressure between the inside and the outside of the cell. The movies were recorded at a speed of approximately 4 frames per second with 1024 × 1024 pixels in BF-TEM imaging. The beam-induced temperature rise is not significant under our typical TEM imaging conditions. (a) The area of the particle was computed with πr2 (b) The volume of the particle was computed with .

The first stage produces 257 images each of size 767 × 767, and 46 images are manually segmented to generate ground truth data. The deep learning algorithm discussed in this section is a supervised neural network and requires ground truth to train the network. The neural network in Figure 1 is presented with image data for training. The data set is split into testing and training in an 80:20 ratio with K-fold data split methods. The network is trained with a stochastic gradient descent algorithm. We used a new loss function with backpropagation and an RMSprop optimizer for parameter adjustment, as described in Loss function section. The optimizer uses a learning rate value of 1 e–5, a weight decay value of 1 e–8, and a momentum value of 0.99. The images are cropped, and the training images only depict the spotlight region. There are two methods tested for cropping the image, as the spotlight is not static and often adjusts during the course of data capture. A circular region detection technique using the Hough circle method is employed in the first approach. The second method uses a K-mean clustering technique proposed previously. The latter approach is computationally expensive but caters to the high variations in the spotlight perturbations. A parallel implementation of the stream of images with a pipeline of cropping and segmentation may reduce the time. The same ground-truth image is used to compute the loss, as the additional edge loss is implemented as a cross-entropy loss with the edges generated from the ground-truth image. At inference time, only the cropped image of the particle is presented to the network, which generates the segmented particle. The morphological measurements are generated using the formulation given in the Results section.

The experimentation of this study is extensively evaluated with quantitative and qualitative results. Two sets of evaluation criteria were adopted for the study. The first set comprises morphology statistics of the particle during its growth, while the second set includes IoU and F1 scores from CV evaluation criteria. Qualitative results are shown as segmentation of the particle and comparison of segmentation images from various techniques. The Au SNP has spikes and the body, and an accurate quantitative measure may be masked by the size of the body; thus, a qualitative evaluation comparison with groundtruth may help the expert for a better comparison. There are 257 frames with the Au SNP images in all for which the evaluation is possible and only 68 groundtruth images are available. The training set uses 50 images, and the remaining are used to generate the quantitative results. However, qualitative results can be given on all 257 images. Figures 2, 3, and 6 show the qualitative results of the segmented images with the proposed methods. Figure 2 shows the comparison of segmented images with three alternatives: a Mask R-CNN, a UNet with edge prior in the input, and a UNet without edge prior. The maskRCNN hardly detects the spike, although it detects the body of the particle and thick protruded spikes. The UNet detects the spike, and with an edge prior in the input it also sharpens the segmented spikes, but it loses the actual length of spikes. The proposed UNet with the boundary-loss expression detects almost all the visible spikes of the particle, and the segmented particle is very close in shape to the hand-segmented groudtruth. Figure 3 further depicts the difference between the techniques tested. The segmented particle shown in blue is placed over the hand-segmented image shown in gray to show the fine-grained comparisons. The proposed UNet trained with an additional boundary-loss term shows the best results and detects the spikes close to the particle perceived by human experts in images. Briefly, the former compared techniques do not recognize the particular spiky structure of the particle, whereas the later proposed techniques show improved segmentation. The proposed network further refines the boundaries of the spikes and improves the overall segmentation results. Figure 4 shows size of particle computed with proposed technique at various stages of its growth. Figure 4a depicts the plot of the area calculated by mapping the slide length of 5 μm to the number of pixels (767 × 767) and the area of the circle enclosing the particle. In contrast, the volume of the sphere formula is used to calculate the volume with prior knowledge of the spherical shape of the Au SNP, and Figure 4b depicts the plot of volume over the course of its life. Furthermore, Table 1 shows the IOU, F1, precision, and recall scores of the images in the test set with the proposed technique and three other neural networks. The compared networks are MaskRCNN UNet with edges prior and UNet without the added loss term. The recall values analyzed in combination with the qualitative results of segmentation show improvement with the proposed methods. Figure 4 shows a plot of particle size over its growth calculated using segmented images of the proposed and compared networks. It is visible in the plot that the hand-segmented images are prone to error considering that the particle does not shrink over the period of its growth. Figure 6 shows the segmentation of the particle at different stages during the period of its growth during the experiment with the proposed UNet and additional loss function. Figure 7 shows the detached spikes in a fully grown Au SNP, and in this particular frame the number of spikes is 32. The height of the biggest spike shown is 0.48 μm (calculated using the same pixel to slide ratio as discussed above). Moreover, Figure 7 shows a graph depicting the number of spikes calculated over the period of the complete experiment. It is important to note that the schematics of the Au SNP as a 3D object are not entirely visible in the 2D electron microscopic images used in this study. The morphology of spikes is an approximation of the only visible side of the particle. This can be extended in the future with anticipated advancements in electron microscopic imaging techniques to capture a complete 3D view of a particle. Finally, we confirm that the plots of particle growth clearly exhibit two different growth regimes: the initial slower growth (called the “mixed regime”, which is limited by the mixture of surface reaction and Au formation rate), followed by faster growth (which is entirely governed by Au formation rate), as made in detail in our previous paper.

Fine segmentation of nanoparticles is desired for the automated morphology operations in electron microscopy images. This study uses a deep neural network with a proposed boundary-favoring loss function to finely segment the spikes of Au SNP in images. The segmented particles are used to automatically generate the statistics of the morphology of the particle over the course of its growth in an experiment. The experimentation shows the proposed methodology performance is on par with the manually calculated statistics of the particle in images. The work can be extended by studying multiple particles on one slide. Moreover, an FPGA implementation of the proposed methodology will help its integration in the electron microscope hardware and with it in camera morphology statics generation.