Naoki Nakanishi,Yoshito Otake,Yuta Hiasa,Yi Gu,Keisuke Uemura,Masaki Takao,Nobuhiko Sugano,Yoshinobu Sato
Decomposition of musculoskeletal structures from radiographs using an improved CycleGAN framework
Radiography,Three-dimensional imaging
This paper presents methods of decomposition of musculoskeletal structures from radiographs into multiple individual muscle and bone structures. While existing solutions require dual-energy scan for the training dataset and are mainly applied to structures with high-intensity contrast, such as bones, we focused on multiple superimposed muscles with subtle contrast in addition to bones. The decomposition problem is formulated as an image translation problem between (1) a real X-ray image and (2) multiple digitally reconstructed radiographs, each of which contains a single muscle or bone structure, and solved using unpaired training based on the CycleGAN framework. The training dataset was created via automatic computed tomography (CT) segmentation of muscle/bone regions and virtually projecting them with geometric parameters similar to the real X-ray images. Two additional features were incorporated into the CycleGAN framework to achieve a high-resolution and accurate decomposition: hierarchical learning and reconstruction loss with the gradient correlation similarity metric. Furthermore, we introduced a new diagnostic metric for muscle asymmetry directly measured from a plain X-ray image to validate the proposed method. Our simulation and real-image experiments using real X-ray and CT images of 475 patients with hip diseases suggested that each additional feature significantly enhanced the decomposition accuracy. The experiments also evaluated the accuracy of muscle volume ratio measurement, which suggested a potential application to muscle asymmetry assessment from an X-ray image for diagnostic and therapeutic assistance. The improved CycleGAN framework can be applied for investigating the decomposition of musculoskeletal structures from single radiographs.
Decomposition of musculoskeletal structures from radiographs using an improved CycleGAN framework

This paper presents methods of decomposition of musculoskeletal structures from radiographs into multiple individual muscle and bone structures. While existing solutions require dual-energy scan for the training dataset and are mainly applied to structures with high-intensity contrast, such as bones, we focused on multiple superimposed muscles with subtle contrast in addition to bones. The decomposition problem is formulated as an image translation problem between (1) a real X-ray image and (2) multiple digitally reconstructed radiographs, each of which contains a single muscle or bone structure, and solved using unpaired training based on the CycleGAN framework. The training dataset was created via automatic computed tomography (CT) segmentation of muscle/bone regions and virtually projecting them with geometric parameters similar to the real X-ray images. Two additional features were incorporated into the CycleGAN framework to achieve a high-resolution and accurate decomposition: hierarchical learning and reconstruction loss with the gradient correlation similarity metric. Furthermore, we introduced a new diagnostic metric for muscle asymmetry directly measured from a plain X-ray image to validate the proposed method. Our simulation and real-image experiments using real X-ray and CT images of 475 patients with hip diseases suggested that each additional feature significantly enhanced the decomposition accuracy. The experiments also evaluated the accuracy of muscle volume ratio measurement, which suggested a potential application to muscle asymmetry assessment from an X-ray image for diagnostic and therapeutic assistance. The improved CycleGAN framework can be applied for investigating the decomposition of musculoskeletal structures from single radiographs.

Quantitative assessments and patient-specific simulations of musculoskeletal systems have been studied extensively in orthopedic surgeries for effective treatment planning and investigation of disease progression. The evaluation of the volume and density of each individual muscle is useful in the diagnosis of muscular dystrophy and sarcopenia, rehabilitation planning, and personalized biomechanics simulation. However, the challenge in the existing method is difficulty in the measurement of muscle mass on a regular basis because three-dimensional (3D) imaging, such as CT and MRI, imposes a high burden on the patient due to radiation exposure in CT, long scan time in MRI, and high cost in both CT and MRI. Radiography exposes patients to far less radiation than CT, making it a more useful modality in routine clinical settings than both CT and MRI. It is thus used for routine clinical checkups of skeletal alignment or joint disease progression, such as knee osteoarthritis. However, the analyses in radiography have been limited to bones. A method for bone decomposition from a radiography image, that is, separation of intensity components due to bones from those due to soft tissues, was developed, and its clinical importance was demonstrated by a significant improvement in bone mineral density estimation accuracy compared with estimation from a non-decomposed X-ray image. However, for training, this method used paired and accurately registered datasets of X-ray images and CT data, which is not only difficult in training data construction but also difficult for the application to muscles because the muscles deform even between radiography and CT imaging. Radiography decomposition into individual muscles and bones using unpaired training data could become a low-cost, low-radiation-dose alternative for frequent monitoring of detailed musculoskeletal conditions, including muscle mass measurement. We are not aware of any method that allows for the decomposition and measurement of muscle mass from a radiography image. One motivating evidence toward addressing muscles in X-ray images in this study is that the edges of some muscles are partly visible to human eyes in local contrast-enhanced X-ray images (Fig. 1). We hypothesized that these X-ray images contain information sufficient for discriminating individual muscles in addition to bones. This study aimed to decompose individual muscle and bone regions from an X-ray image. We formulated the problem as an image translation from a real X-ray image to a set of decomposed images of individual muscles and bones, and we solved it using generative adversarial networks (GANs). As a training dataset, we used muscle and bone regions in CT automatically segmented using Bayesian U-net and generated digitally reconstructed radiographs (DRRs) of each muscle or bone combined with real X-ray images.

Most related works can be classified into segmentation and decomposition as summarized in Table 1. The pixel-wise classification is called segmentation, whereas the pixel-wise estimation of multiple continuous attenuation values of individual structures that compose each X-ray’s line integral (i.e., the sum of X-ray attenuation along one X-ray beam) is called decomposition.

Novikov et al., Chen et al., and Eslami et al. proposed deep learning-based segmentation methods for one of (or all of) the lung, heart, and clavicle regions. They used manually annotated label images of the target regions on the X-ray images, which incur a high annotation cost. Zhang et al. employed DRRs created from CT images. Their method is advantageous because the diverse training dataset could be automatically generated by perturbing translation and rotation of the CT. However, the integration of attenuation was not considered in the above studies; thus, decomposition was not realized.

One classical machine learning approach for X-ray image decomposition used the training data set acquired using dual-energy X-ray absorptiometry (DXA). Suzuki et al. and Yang et al. performed bone suppression (i.e., bone tissue decomposition) from chest X-ray images. While the focus in these studies was enhancing lung lesions covered by rib bones, the same approach does not apply to the decomposition of multiple soft tissues due to the limitation of discriminative ability in DXA acquisition. Li et al. proposed an X-ray image decomposition method for bone suppression by converting chest X-ray images into DRRs of bone and lung regions. However, quantitative evaluation of the decomposed DRRs with the real X-ray images has not been performed. In addition to bones, Albarqouni et al. performed the decomposition of multiple organs from X-ray images. The decomposition model was trained by pairing X-ray image-like DRRs created by projecting CT images as inputs and a series of decomposed DRRs created using CT images and mask images as outputs. They introduced a new loss term, reconstruction loss, which considers that multiple regions are superimposed to “reconstruct” the original X-ray image. The proposed reconstruction loss was tested only on DRRs generated from calibrated CT images in an ideal condition; the method was not evaluated on real X-ray images. Gu et al. recently proposed a method for quantitatively decomposing a partial bone region, namely, the proximal femur region, from an X-ray image to measure bone mineral density. Howerver, this work was also limited to bones. To the best of our knowledge, no study has considered the following two features, which are essential for real clinical applications in musculoskeletal diagnosis. Decomposition of multiple soft tissues (i.e., muscles) in addition to bones Quantitative evaluations of decompositions from real X-ray images Image translation using GANs has been extensively studied and shown to outperform conventional methods. However, in our context of muscle tissue decomposition from X-ray image using DRRs created from CT, obtaining a paired training dataset (i.e., input and output images are registered to each other) was difficult due to the shape changes (even slightly) of flexible muscles in different poses between CT and X-ray image scans. Therefore, algorithms that can handle unpaired training data, such as cycle-consistent GAN (CycleGAN), are our focus. Several recent works proposed an improved version of CycleGAN and compared the performance using the original CycleGAN as the baseline. Hiasa et al. introduced a new loss term in CycleGAN, the gradient correlation (GC) loss, which preserves the alignment of edges by calculating the correlation of gradients in input and output images. Wang et al. proposed an improvement of the Pix2Pix model, a commonly employed GAN model for image translation that uses a paired training dataset by incorporating hierarchical learning (HL) for high-resolution image translation. In this study, we integrated the three previously proposed features, i.e., the reconstruction loss, GC loss, and HL, into the CycleGAN framework to achieve edge-alignment preserved and high-resolution image decomposition of real X-ray images. The contribution of this study is three-fold. (1) Proposal of methods to decompose multiple soft tissues (i.e., muscles) and bones from real X-ray images. (2) Integration of the three previously proposed features in different contexts into CycleGAN to achieve high-resolution stable image translation robust against intensity variation in real X-ray images. (3) Thorough evaluations of the proposed methods using a dataset comprising real X-ray images and CT images of 475 patients, including evaluations with a metric on muscle asymmetry.

The overview of the methods is shown in Fig. 2. The training dataset comprises real X-ray images and their associated DRRs. The DRRs of individual muscle and bone regions are created using CT images and their associated 3D muscle/bone masks, which are automatically segmented from CT. In the following subsections, we describe the projection model used in this study, proposed methods for the decomposition of X-ray images, implementation details, and evaluation metrics.

Our projection model assumes a point X-ray source (i.e., pinhole camera model) and one virtual X-ray beam per detector element, which ignores the effect of the finite focal spot size and finite area of the detector pixel. Furthermore, the pixel intensity at two-dimensional (2D) position in an X-ray image is approximated as a line integral of a linear attenuation coefficient (AC) derived from a CT value along a ray penetrating an object, where the scattering and beam hardening effects are ignored. We consider that the abovementioned assumptions are reasonable to well-approximate a virtual X-ray image in light of many previous studies. We assume that an X-ray image is regarded as the superimposition of the virtual projection (DRR) images of K individual muscle/bone regions () and the background, i.e., the regions other than the K regions (e.g., fat, pelvic viscera, and vessels), denoted as . Thus, is approximated bywhere d denotes the distance between the X-ray source and detector, denotes the one-dimensional (1D) profile of the AC along a ray penetrating the CT image, which corresponds to the pixel position on the 2D projection plane, s is an arc length parameter on the ray, and is the 1D binary profile of the segmentation mask along the ray corresponding to of the n-th muscle/bone region in the CT image. The segmentation masks were automatically obtained from CT images using Bayesian U-net. We used the distance between the source and the detector recorded in the DICOM header of the X-ray image (the attribute named “Distance Source to Detector”) for generating DRRs. To obtain an approximate patient position with respect to the X-ray imager coordinate system, each CT image was registered to the X-ray image of the same patient using a rigid 2D–3D registration algorithm proposed in. Notably, owing to the hip angle difference and deformation of muscles between a CT image (acquired in supine position) and an X-ray image (acquired in standing position), the rigid registration only roughly aligned overall structures, which is the reason we required an unpaired image translation algorithm, we explain next.

For an unpaired image translation from a real X-ray image to a set of virtual projection images of the individual muscle/bone regions , the CycleGAN proposed by Zhu et al. was employed. Figure 2b shows the workflow of the proposed image translation. The generators and , generate from and from , respectively. The discriminators and discriminate whether the image is real or fake. The adversarial loss, , that we used in this study is defined as follows:wherein which x and y are images from domains and , respectively. The cycle consistency loss is given by One challenge in the generation of a high-resolution image such as an X-ray image is to ensure stability. Because a higher resolution makes it easier to differentiate generated and training images, the discriminator converges faster than the generator, making the training unstable. Therefore, we incorporate an approach using multiple resolution training images which is inspired by HL proposed for paired image translation into our unpaired image translation. The network structure is shown in Fig. 2c. The weight parameters for the generators, and , are trained with the low-resolution images first and subsequently with the high-resolution images. We employ a loss term that we call reconstruction GC loss to match the gradient in the sum of the decomposed images with the gradient of the input X-ray image. Albarqouni et al. defined the reconstruction loss term as the -loss between the sum of decomposed images and input images. Although -loss works in a simulation experiment as in where the input image was also DRRs generated from CT image, it will not be straightforward to apply the same loss term directly to real X-ray images because of the discrepancy from the ideal projection due to various physical phenomena, including the variation in scan protocol (e.g., X-ray tube current, and voltage), beam hardening, scattering, blurring due to focal spot size, and postprocessing filters built in the imaging system (e.g., bone enhancement filter). To improve robustness against the variation in absolute intensity, we incorporate gradient-matching constraints proposed in, which use the GC in the regularization term. The GC between the input image and reconstructed by summation of each decomposed image is used in the regularization term, which we named the reconstruction GC loss and defined as follows:where represents the sum of the multichannel output of the generator , represents one channel from the multichannel DRR. GC(A, B) is given bywhere NCC(A, B) denotes the normalized cross-correlation of A and B, and and are, respectively, u and v components of the gradient vector. Finally, the total loss function, , is formulated as follows:where and are the hyperparameters determining the balance among the three terms.

The hierarchical CycleGAN incorporating the reconstruction GC loss was trained by solving the mini-max problem, given byusing Adam with a learning rate of 0.0002 for the first 100 epochs and linearly decreasing to 0 for the later 100 epochs. The generator was trained hierarchically with nine residual blocks, and the discriminator was PatchGAN. The balance parameters were experimentally determined to be and . The average training time for the conventional and proposed methods was approximately 8.0 and 9.7 h, respectively (2.7 and 7.0 h for the training of global () and local () mappings, respectively), on a workstation equipped with an Intel Xeon processor (2.30 GHz, 4 cores) with NVIDIA Titan RTX (24 GB memory). The average computation time for the inference for one case was about 2.5 s, excluding the file access.

We propose the volume ratio of left and right muscles as an evaluation metric, which will be potentially used as a diagnostic metric to evaluate muscle asymmetry directly measured from a plain X-ray image. As described later in “Data set”, the patients of the dataset used in the experiments were before undergoing total hip arthroplasty, and one side, the operation side, was more affected than the other side. Therefore, the ratio of the affected/unaffected volumes, / of muscle , denoted by , is used for the evaluation. Our measurement of the volume ratio using a single 2D projection image is based on a parallel projection model (i.e., all rays penetrate the volume in parallel). The sum of line integrals within the muscle region of an affected or unaffected side in a 2D DRR is given bywhere V denotes the 3D volume of the affected or unaffected side of the target muscle and denotes the average AC within its region. Equation (8) indicates that the sum of the intensity value in can be approximated as a constant multiple of volume V of each muscle . Therefore, the ratio of the volumes, in particular, AC-weighted volumes, of muscles between the affected and unaffected sides, (hereinafter “affected/unaffected volume ratio”), can be estimated only from the decomposed DRR images bywhere and are the DRR and average AC of the affected side, respectively, and and are those of the unaffected side, respectively. Its accuracy evaluation is demonstrated later in “Real image experiments” by comparison with the volume ratio obtained from the 3D mask of each muscle, which was automatically segmented from CT. According to, around 20% decrease in the volume ratio and around 10 HU decrease in the CT value difference occur on average between the unaffected and affected sides. The 10 HU decrease causes around 1% decrease in the average AC ratio. Therefore, the main factor of will be the volume ratio, and the effects of the average AC ratio will be small. Thus, is approximately viewed as the volume ratio.

In addition to the diagnostic metric described above, two general evaluation metrics were employed in the experiments—the peak signal-to-noise ratio (PSNR) and Dice coefficient (DC)—for evaluating the absolute intensity and silhouette shape, respectively. To calculate DC, the decomposed image and ground truth (i.e., DRR generated from a CT image) were binarized. For the binarization threshold selection, we computed the average DC over the empirically determined threshold range of the intensity which was visually confirmed to obtain appropriate segmented regions of the target structures. To suppress the bias due to the threshold selection, we selected the threshold value for each method, by which the best average DC was attained. For statistical analysis, the Friedman test was employed to compare results for each group (in our case, the target structures were grouped into three: bones, hip muscles, and thigh muscles). The Wilcoxon test with Bonferroni correction for multiple comparisons was used.

This study was approved by the Institutional Review Board at Osaka University (No.15056-3) and Nara Institute of Science and Technology (No.2019-M-6), and written informed consent was waived because of the retrospective design. All the following procedures were in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. The summary of the data set used in this study is shown in Table 2. The data set consists of the lower extremity X-ray images of standing posture and the CT images of 475 patients (69 males and 406 females) acquired for the surgical planning of total hip arthroplasty at Osaka University Hospital, Suita, Japan. Ideally, the X-ray image and CT can be paired in a patient-wise manner, however, in practice, accurate 2D–3D registration of the muscles is quite difficult due to weak and incomplete edges of the muscles in X-ray images and nonrigid deformation with discontinuities between different muscles. Therefore, we used a CycleGAN-based framework of unpaired image translation. In order to construct the training data set, the CT images were automatically segmented with the Bayesian U-net. The CT intensity in the unit of HU was first converted to a linear AC using the linear relationship assuming the monoenergetic X-ray. The muscle/bone regions segmented in this study were the 23 regions (3 bones, 19 muscles, and all other tissues as one object, which we call background) as shown in Fig. 3. The intensity values were normalized to [0,255] using the average of the maximum and minimum values of in the training cases. Since the normalization range was fixed throughout all cases, the decomposed DRRs synthesized by the proposed method can be denormalized to the original intensity range, which allows comparing the absolute density between cases. and were resized to using the bi-cubic interpolation. The relatively small image size is due primarily to the CT resolution and GPU memory limitations. The voxel size of our CT was mm, as shown in Table 2, which limited the resolution of DRR. The reason for resizing the radiograph to 512 was to align it with the image size of the DRR (although we consider that it would be a possible future work to explore the efficient use of the high-resolution content in the radiograph by applying a super-resolution algorithm to DRRs).

Two-fold cross-validation was performed in both simulation and real image experiments. The patient-wise split (237 and 238 patients) was performed. Twelve cases in the training dataset were employed for the validation. The following four methods, including two conventional and two proposed methods summarized in Table 3, were compared. Conventional method #1 refers to the original CycleGAN without any additional features with generators of nine residual blocks and a discriminator of PatchGAN. Conventional method #2 refers to a method combining Conventional method #1 and the reconstruction loss proposed by Albarqouni et al. with a weight balance parameter of 0.5. Proposed method #1 used HL with the same generator as Conventional method #1. Proposed method #2 combined Proposed method #1 and the reconstruction GC loss. Regarding the reconstruction loss, Li et al. and Zhang et al. noted that since it directly measures the difference between the virtual (i.e., DRR) and real X-ray images without taking into account their domain gap, it works only for a few real chest X-ray images with structures similar to the DRRs (i.e., when the domain gap is small). Therefore, the reconstruction loss was not included in the proposed methods. Detail of the parameters used in the experiments were summarized in Table 4

In the simulation experiments, input images were DRRs generated from CT images with additional Poisson noise, simulating the quantum noise appearing in real X-ray images (Fig. 4), which we denote by . Projection parameters were set as described in “Projection model”.

Real X-ray images acquired with the actual clinical protocols were used. The evaluation was performed in two steps. (1) For rigid structures, namely pelvis and femur bones, PSNR and DC were evaluated using the DRRs obtained from the CT images registered to the X-ray images using a rigid 2D–3D registration as the ground truth. (2) For nonrigid structures, namely, muscles, the ground truth DRR (i.e., with the target muscles nonrigidly registered to that in real X-ray images) was difficult to obtain. Instead, the volume ratio of the (more) affected and (more) unaffected sides, introduced in “Muscle asymmetry metric”, was evaluated. An expert physician defined the affected side. We targeted nine muscles eliminating those not visible or only partially visible in the X-ray image’s field of view (FOV). Red underlines in Fig. 3 show the evaluated muscles.

Table 5 shows the summary of the experimental results.

The PSNR and DC for the four methods summarized in Table 3, using the simulated X-ray images (DRRs) (Fig.4 and “Simulation experiments”) generated by projecting the CT image, are shown in Fig. 5. The illustrative decomposition results are shown in Fig. 6. The accuracy of the decomposition results except for the background region was quantitatively evaluated. The muscle and bone regions were grouped into three groups—bone, hip muscle, and thigh muscle regions (Fig. 3), and PSNR and DC were averaged over each group for each case. The means and standard deviations of PSNR and DC of the four methods are shown in Fig. 5. Significant improvements were observed in Proposed methods #1 and #2 compared with Conventional methods #1 and #2 in both PSNR and DC. Further, Proposed method #2 showed significant improvements in PSNR and DC compared with Proposed method #1 except for thigh muscles. As shown in Fig. 6, higher accuracy in Proposed method #2 was visually observed in the femoral head decomposition results, indicating that the combination of the HL and reconstruction GC loss in Proposed method #2 was especially effective in bone regions with higher contrast than muscles. Please see Table S1 in Supplementary Materials for the complete ablation study where the three features (i.e., local enhancer in generator, reconstruction loss, and reconstruction GC loss) were excluded one at a time. The plot of PSNR as a function of the number of epochs in one example training trial is shown in Figure S1 in Supplementary Materials.

The quantitative evaluation results of the pelvis and femur regions decomposed from real X-ray images are shown in Fig. 7. The real image experiments also showed significant improvements in Proposed methods #1 and #2 compared with Conventional methods #1 and #2 in both PSNR and DC. Further, Proposed method #2 showed significant improvements in PSNR and DC compared with Proposed method #1 in both the pelvis and femur. Figure 8 shows the comparison of affected/unaffected volume ratio (horizontal axis) measured from the real X-ray images of the nine evaluated muscles against the ground truth obtained from CT (vertical axis). Proposed method #2 tended to show a higher intraclass correlation coefficient (ICC) for most muscles than other methods (Fig. 8c). Figure 9 illustrates representative results for the femur and gluteus maximus muscles. Proposed method #2 demonstrated superior performance, especially in the proximal femur, indicated in the red rectangle, in delineating the structure boundaries (yellow arrows in Fig. 9) than the other methods. Figure 10 shows representative failure cases. The failure in patient #396 was attributed to the abnormal femur position, whereas that in patient #235 was attributed to excessive obesity compared with other patients in the training data set.

We presented methods to decompose an X-ray image of the hip into projection images each of which contains individual muscle and bone structures. We experimentally validated that the proposed CycleGAN-based image translation method with a combination of HL and reconstruction GC loss significantly improved the decomposition accuracy of 3 bones and 19 muscle structures compared with conventional methods. We showed that the quantitative decomposition achieved using the proposed methods, in which the intensity of the decomposed images has the unit equivalent to the line integral of the ACs, allowed evaluation of muscle asymmetry in case of relatively large muscles, which is clinically relevant in diagnosis and therapeutic planning in orthopedic surgery. To the best of our knowledge, this is the first peer-reviewed article that reports the quantitative validation of individual muscle decomposition from real X-ray images. One advantage of the proposed methods is that automatically segmented 3D masks in CT were employed to construct the training dataset. We showed that automatic segmentation in 3D could be leveraged to construct a relatively large (475 cases) annotated training dataset without manual traces, which was used to train networks for decomposing individual muscles and bones from 2D X-ray images. Our experiments evaluated the effects of each feature that we added to the original CycleGAN (Conventional method #1) independently. In the femur decomposition shown in Fig. 9, for example, the proximal femur region in the image decomposed by the original CycleGAN showed blurry and incorrect shapes at the femoral head and greater trochanter. By adding the reconstruction loss (Conventional method #2), the intensity patterns were better reconstructed but still blurry. The HL alone (Proposed method #1) provided a higher-resolution decomposition, removing the blurriness. By combining the reconstruction GC loss and the HL (Proposed method #2), the decomposed image better reproduced the intensity patterns, and the boundary of the greater trochanter became closer to the ground truth DRR, although the boundary of the femoral head, which is largely deformed due to osteoarthritis, was still not very close. Results shown in Figs. 7 and 9 validate the simulation experiments in Figs. 5 and 6 and support that the proposed methods, especially Proposed method #2 (incorporating the reconstruction GC loss and HL), are superior in the quantitative decomposition of the pelvis/femur regions in real X-ray images. Our added features to CycleGAN were also shown to be effective for quantitative decomposition of individual muscles from real X-ray images. We demonstrated that the proposed methods showed optimal accuracy in estimating the affected/unaffected volume ratio of individual muscles (Fig. 8). Sufficiently high accuracy was obtained for large muscles such as gluteus maximus and gluteus medius; accuracy was not enough for small muscles. One contribution is that we demonstrated the potential usefulness of quantitative muscle volume analysis from plain X-ray images. The two representative failure cases (Fig. 10) that showed low PSNR and DC in the pelvis and femur regions (green and yellow dots in Fig. 7) illustrated the limitation of the learning-based approach, where the cases that were not within the training data distribution failed. One of the failures (Patient #396) is attributed to the over flexion of the hip joint, likely due to inability to stand upright because of the flexion contracture, whereas the other failure (Patient #235) was due to excessive obesity that created skin surface edges different from most training data. One potential technique to mitigate this type of failure is data augmentation. For this purpose, simulations of muscle deformation accompanying with joint motion and simulations of body weight changes would need to be incorporated in data augmentation, which is in our future work. In computer-assisted diagnosis in general, inaccurate prediction of the decomposition in a clinical situation is an important consideration. Hiasa et al. proposed a practical uncertainty metric as a surrogate inaccuracy measure for muscle segmentation in CT, which clinicians can use in clinical diagnosis. A similar approach, we believe, can be used in our radiography-based decomposition. It could also be extended to a more advanced image translation algorithm such as an uncertainty-aware translation. One alternative solution to the real X-ray image decomposition problem is to divide the pipeline into two consecutive neural networks similar to the one proposed by Li et al. in the chest X-ray application, where (1) translation from a real X-ray image to an X-ray image-like DRR is trained with unpaired data sets and then (2) translation from the DRR to decomposed DRRs is trained with paired data sets. Unlike our one-step approach that combines these two steps into one network, the advantage would be in the second step, which allows the data augmentation by adding arbitrary geometric transform to CT, making the model more robust to various rigid misalignment, in exchange for instability and accumulation of errors in the training process of two GANs. A detailed comparison of the two approaches regarding the accuracy, flexibility to data augmentation, and training stability is left as future work.

Using a large number of automatically segmented CT images as a training dataset, this study demonstrated the capability of decomposing the intensity distribution of individual muscles and bones from a single X-ray image. Despite some failures due to large differences from the training sample distribution, this study indicated that the volume ratio between the left and right muscles can be quantified to intraclass correlation coefficients of 0.7 with the ground truth. These findings support future research into frequent monitoring of musculoskeletal health for early detection of sarcopenia and osteoporosis.

Supplementary Information 1.