Question: Represent the article for finding a single-sentence TLDR.

Question: Separating mixed distributions is a long standing challenge for machine learning and signal processing. Applications include: single-channel multi-speaker separation (cocktail party problem), singing voice separation and separating reflections from images. Most current methods either rely on making strong assumptions on the source distributions (e.g. sparsity, low rank, repetitiveness) or rely on having training samples of each source in the mixture. In this work, we tackle the scenario of extracting an unobserved distribution additively mixed with a signal from an observed (arbitrary) distribution. We introduce a new method: Neural Egg Separation - an iterative method that learns to separate the known distribution from progressively finer estimates of the unknown distribution. In some settings, Neural Egg Separation is initialization sensitive, we therefore introduce GLO Masking which ensures a good initialization. Extensive experiments show that our method outperforms current methods that use the same level of supervision and often achieves similar performance to full supervision. Humans are remarkably good at separating data coming from a mixture of distributions, e.g. hearing a person speaking in a crowded cocktail party. Artificial intelligence, on the the hand, is far less adept at separating mixed signals. This is an important ability as signals in nature are typically mixed, e.g. speakers are often mixed with other speakers or environmental sounds, objects in images are typically seen along other objects as well as the . Understanding mixed signals is harder than understanding pure sources, making source separation an important research topic. Mixed signal separation appears in many scenarios corresponding to different degrees of supervision. Most previous work focused on the following settings:Full supervision: The learner has access to a training set including samples of mixed signals {y i} ∈ Y as well as the ground truth sources of the same signals {b i} ∈ B and {x i} ∈ X (such that y i = x i + b i). Having such strong supervision is very potent, allowing the learner to directly learn a mapping from the mixed signal y i to its sources (x i, b i). Obtaining such strong supervision is typically unrealistic, as it requires manual separation of mixed signals. Consider for example a musical performance, humans are often able to separate out the different sounds of the individual instruments, despite never having heard them play in isolation. The fully supervised setting does not allow the clean extraction of signals that cannot be observed in isolation e.g. music of a street performer, car engine noises or reflections in shop windows. The learner has access to a training set containing samples from the mixed signal {y i} ∈ Y as well as samples from all source distributions {b j} ∈ B and {x k} ∈ X. The learner however does not have access to paired sets of the mixed and unmixed signal ground truth (that is for any given y i in the training set, b i and x i are unknown). This supervision setting is more realistic than the fully supervised case, and occurs when each of the source distributions can be sampled in its pure form (e.g. we can record a violin and piano separately in a studio and can thus obtain unmixed samples of each of their distributions). It is typically solved by learning to separate synthetic mixtures b j + x k of randomly sampled b j and x k.No supervision: The learner only has access to training samples of the mixed signal Y but not to sources B and X. Although this settings puts the least requirements on the training dataset, it is a hard problem and can be poorly specified in the absence of strong assumptions and priors. It is generally necessary to make strong assumptions on the properties of the component signals (e.g. smoothness, low rank, periodicity) in order to make progress in separation. This unfortunately severely limits the applicability of such methods. In this work we concentrate on the semi-supervised setting: unmixing of signals in the case where the mixture Y consists of a signal coming from an unobserved distribution X and another signal from an observed distribution B (i.e. the learner has access to a training set of clean samples such that {b j} ∈ B along with different mixed samples {y i} ∈ Y). One possible way of obtaining such supervision, is to label every signal sample by a label, indicating if the sample comes only from the observed distribution B or if it is a mixture of both distributions B + X. The task is to learn a parametric function able to separate the mixed signal y i ∈ Y into sources x i ∈ X and b i ∈ B s.t. y i = b i + x i. Such supervision is much more generally available than full supervision, while the separation problem becomes much simpler than when fully unsupervised. We introduce a novel method: Neural Egg Separation (NES) -consisting of i) iterative estimation of samples from the unobserved distribution X ii) synthesis of mixed signals from known samples of B and estimated samples of X iii) training of separation regressor to separate the mixed signal. Iterative refinement of the estimated samples of X significantly increases the accuracy of the learned masking function. As an iterative technique, NES can be initialization sensitive. We therefore introduce another method -GLO Masking (GLOM) -to provide NES with a strong initialization. Our method trains two deep generators end-to-end using GLO to model the observed and unobserved sources (B and X). NES is very effective when X and B are uncorrelated, whereas initialization by GLOM is most important when X and B are strongly correlated such as e.g. separation of musical instruments. Initialization by GLOM was found to be much more effective than by adversarial methods. Experiments are conducted across multiple domains (image, music, voice) validating the effectiveness of our method, and its superiority over current methods that use the same level of supervision. Our semi-supervised method is often competitive with the fully supervised baseline. It makes few assumptions on the nature of the component signals and requires lightweight supervision. Source separation: Separation of mixed signals has been extensively researched. In this work, we focus on single channel separation. Unsupervised (blind) single-channel methods include: ICA BID3 and RPCA BID10. These methods attempt to use coarse priors about the signals such as low rank, sparsity or non-gaussianity. HMM can be used as a temporal prior for longer clips BID25, however here we do not assume long clips. Supervised source separation has also been extensively researched, classic techniques often used learned dictionaries for each source e.g. NMF BID30. Recently, neural network-based gained popularity, usually learning a regression between the mixed and unmixed signals either directly BID11 or by regressing the mask BID29 BID32. Some methods were devised to exploit the temporal nature of long audio signal by using RNNs BID19, in this work we concentrate on separation of short audio clips and consider such line of works as orthogonal. One related direction is Generative Adversarial Source Separation BID27 BID28 ) that uses adversarial training to match the unmixed source distributions. This is needed to deal with correlated sources for which learning a regressor on synthetic mixtures is less effective. We present an Adversarial Masking (AM) method that tackles the semi-supervised rather than the fully supervised scenario and overcomes mixture collapse issues not present in the fully supervised case. We found that non-adversarial methods perform better for the initialization task. The most related set of works is semi-supervised audio source separation BID26 BID0, which like our work attempt to separate mixtures Y given only samples from the distribution of one source B. Typically NMF or PLCA (which is a similar algorithm with a probabilistic formulation) are used. We show experimentally that our method significantly outperforms NMF.Disentanglement: Similarly to source separation, disentanglement also deals with separation in terms of creating a disentangled representation of a source signal, however its aim is to uncover latent factors of variation in the signal such as style and content or shape and color e.g. BID4; BID7. Differently from disentanglement, our task is separating signals rather than the latent representation. Generative Models: Generative models learn the distribution of a signal directly. Classical approaches include: SVD for general signals and NMF BID17 ) for non-negative signals. Recently several deep learning approaches dominated generative modeling including: GAN BID6, VAE BID15 and GLO BID1. Adversarial training (for GANs) is rather tricky and often leads to mode-collapse. GLO is non-adversarial and allows for direct latent optimization for each source making it more suitable than VAE and GAN. In this section we present our method for separating a mixture of sources of known and unknown distributions. We denote the mixture samples y i, the samples with the observed distribution b i and the samples from the unobserved distribution x i. Our objective is to learn a parametric function T , such that b i = T (y i).Full Supervision: In the fully supervised setting (where pairs of y i and b i are available) this task reduces to a standard supervised regression problem, in which a parametric function T (typically a deep neural network) is used to directly optimize: DISPLAYFORM0 Where typically is the Euclidean or the L 1 loss. In this work we use = L 1 .Mixed-unmixed pairs are usually unavailable, but in some cases it is possible to obtain a training set which includes unrelated samples x j and b k e.g. BID29 BID32. Methods typically randomly sample x j and b k sources and synthetically create mixtures y jk = x j + b k. The synthetic pairs (b k, y jk) can then be used to optimize Eq. 1. Note that in cases where X and B are correlated (e.g. vocals and instrumental accompaniment which are temporally dependent), random synthetic mixtures of x and b might not be representative of y and fail to generalize on real mixtures. Semi-Supervision: In many scenarios, clean samples of both mixture components are not available. Consider for example a street musical performance. Although crowd noises without street performers can be easily observed, street music without crowd noises are much harder to come by. In this case therefore samples from the distribution of crowd noise B are available, whereas the samples from the distribution of the music X are unobserved. Samples from the distribution of the mixed signal Y i.e. the crowd noise mixed with the musical performance are also available. The example above illustrates a class of problems for which the distribution of the mixture and a single source are available, but the distribution of another source is unknown. In such cases, it is not possible to optimize Eq. 1 directly due to the unavailability of pairs of b and y. Neural Egg Separation: Fully-supervised optimization (as in Eq. 1) is very effective when pairs of b i and y i are available. We present a novel algorithm, which iteratively solves the semi-supervised task as a sequence of supervised problems without any clean training examples of X. We name the method Neural Egg Separation (NES), as it is akin to the technique commonly used for separating egg whites and yolks. The core idea of our method is that although no clean samples from X are given, it is still possible to learn to separate mixtures of observed samples b j from distribution B combined with some estimates of the unobserved distribution samplesx i. Synthetic mixtures are created by randomly sampling an approximate samplex i from the unobserved distribution and combining with training sample b j: DISPLAYFORM1 thereby creating pairs (ỹ ij, b j) for supervised training. Note that the distribution of synthetic mixturesỹ ij might be different from the real mixture sample distribution y j, but the assumption (which is empirically validated) is that it will eventually converge to the correct distribution. During each iteration of NES, a neural separation function T is trained on the created pairs by optimizing the following term: DISPLAYFORM2 At the end of each iteration, the separation function T can be used to approximately separate the training mixture samples y i into their sources: DISPLAYFORM3 The refined X domain estimatesx i are used for creating synthetic pairs for finetuning T in the next iteration (as in Eq. 3).The above method relies on having an estimate of the unobserved distribution samples as input to the first iteration. One simple scheme is to initialize the estimates of the unobserved distribution samples in the first iteration asx i = c · y i, where c is a constant fraction (typically 0.5). Although this initialization is very naive, we show that it achieves very competitive performance in cases where the sources are independent. More advanced initializations will be discussed below. At test time, separation is simply carried out by a single application of the trained separation function T (exactly as in Eq. 4). Mixture samples {y i}, Observed source samples {b j} Result: Separation function T Initialize synthetic unobservable samples withx i ← c · y i or using AM or GLOM; Initialize T with random weights; DISPLAYFORM0 Optimize separation function for P epochs: DISPLAYFORM1 Update estimates of unobserved distribution samples: DISPLAYFORM2 Algorithm 1: NES Algorithm Our full algorithm is described in Alg. 1. For optimization, we use SGD using ADAM update with a learning rate of 0.01. In total we perform N = 10 iterations, each consisting of optimization of T and estimation ofx i, P = 25 epochs are used for each optimization of Eq. 3.GLO Masking: NES is very powerful in practice despite its apparent simplicity. There are some cases for which it can be improved upon. As with other synthetic mixture methods, it does not take into account correlation between X and B e.g. vocals and instrumental tracks are highly related, whereas randomly sampling pairs of vocals and instrumental tracks is likely to synthesize mixtures quite different from Y. Another issue is finding a good initialization-this tends to affect performance more strongly when X and B are dependent. We present our method GLO Masking (GLOM), which separates the mixture by a distributional constraint enforced via GLO generative modeling of the source signals. GLO BID1 ) learns a generator G, which takes a latent code z b and attempts to reconstruct an image or a spectrogram: b = G(z b). In training, GLO learns end-to-end both the parameters of the generator G as well as a latent code z b for every training sample b. It trains per-sample latent codes by direct gradient descent over the values of z b (similar to word embeddings), rather than by a feedforward encoder used by autoencoders (e.g. z b = E(b)). This makes it particularly suitable for our scenario. Let us define the set of latent codes: DISPLAYFORM3 The optimization is therefore: DISPLAYFORM4 We propose GLO Masking, which jointly trains generators: G B for B and G X for X such that their sum in mixture samples y = G B (z DISPLAYFORM5 We use the supervision of the observed source B to train G B , while the mixture Y contributes residuals that supervise the training of G X . We also jointly train the latent codes for all training images: z b ∈ Z for all b ∈ B, and z B y ∈ Z B, z X y ∈ Z X for all y ∈ Y. The optimization problem is: DISPLAYFORM6 As GLO is able to overfit arbitrary distributions, it was found that constraining each latent code vector z to lie within the unit ball z · z ≤ 1 is required for generalization. Eq. 6 can either be optimized end-to-end, or the left-hand term can be optimized first to yield Z, G B , then the right-hand term is optimized to yield Z B, Z X, G X . Both optimization procedures yield similar performance (but separate training does not require setting λ). Once G B and G X are trained, for a new mixture sample we infer its latent codes: DISPLAYFORM7 Our estimate for the sources is then: DISPLAYFORM8 Masking Function: In separation problems, we can exploit the special properties of the task e.g. that the mixed signal y i is the sum of two positive signals x i and b i. Instead of synthesizing the new sample, we can instead simply learn a separation mask m, specifying the fraction of the signal which comes from B. The attractive feature of the mask is always being in the range (in the case of positive additive mixtures of signals). Even a constant mask will preserve all signal gradients (at the cost of introducing spurious gradients too). Mathematically this can be written as: DISPLAYFORM9 For NES (and baseline AM described below), we implement the mapping function T (y i) using the product of the masking function y i · m(y i). In practice we find that learning a masking function yields much better than synthesizing the signal directly (in line with other works e.g. BID29 ; BID5).GLOM models each source separately and is therefore unable to learn the mask directly. Instead we refine its estimate by computing an effective mask from the element-wise ratio of estimated sources: DISPLAYFORM10 Initializing Neural Egg Separation by GLOM: Due to the iterative nature of NES, it can be improved by a good initialization. We therefore devise the following method: i) Train GLOM on the training set and infer the mask for each mixture. This is operated on images or mel-scale spectrograms at 64 × 64 resolutions ii) For audio: upsample the mask to the resolution of the highresolution linear spectrogram and compute an estimate of the X source linear spectrogram on the training set iii) Run NES on the observed B spectrograms and estimated X spectrograms. We find experimentally that this initialization scheme improves NES to the point of being competitive with fully-supervised training in most settings. To evaluate the performance of our method, we conducted experiments on distributions taken from multiple real-world domains: images, speech and music, in cases where the two signals are correlated and uncorrelated. We evaluated our method against 3 baseline methods:Constant Mask (Const): This baseline uses the original mixture as the estimate. This baseline method, proposed by BID26, first trains a set of l bases on the observed distribution samples B by Sparse Adversarial Masking (AM): As an additional contribution, we introduce a new semi-supervised method based on adversarial training, to improve over the shallow NMF baseline. AM trains a masking function m so that after masking, the training mixtures are indistinguishable from the distribution of source B under an adversarial discriminator D. The loss functions (using LS-GAN BID18) are given by: DISPLAYFORM0 Differently from CycleGAN BID34 and DiscoGAN BID14, AM is not bidirectional and cannot use cycle constraints. We have found that adding magnitude prior L 1 (m(y), 1) improves performance and helps prevent collapse. To partially alleviate mode collapse, we use Spectral Norm on the discriminator. We evaluated our proposed methods:GLO Masking (GLOM): GLO Masking on mel-spectrograms or images at 64 × 64 resolution. The NES method detailed in Sec. 3. Initializing X estimates using a constant (0.5) mask over Y training samples. Initializing NES with the X estimates obtained by GLO Masking. To upper bound the performance of our method, we also compute a fully supervised baseline, for which paired data of b i ∈ B, x i ∈ X and y i ∈ Y are available. We train a masking function with the same architecture as used by all other regression methods to directly regress synthetic mixtures to unmixed sources. This method uses more supervision than our method and is an upper bound. More implementation details can be found in appendix A. In this section we evaluate the effectiveness of our method on image mixtures. We conduct experiments both on the simpler MNIST dataset and more complex Shoes and Handbags datasets. To evaluate the quality of our method on image separation, we design the following experimental protocol. We split the MNIST dataset BID16 into two classes, the first consisting of the digits 0-4 and the second consisting of the digits 5-9. We conduct experiments where one source has an observed distribution B while the other source has an unobserved distribution X. We use 12k B training images as the B training set, while for each of the other 12k B training images, we randomly sample a X image and additively combine the images to create the Y training set. We evaluate the performance of our method on 5000 Y images similarly created from the test set of X and B. The experiment was repeated for both directions i.e. 0-4 being B while 5-9 in X, as well as 0-4 being X while 5-9 in B.In Tab. 1, we report our on this task. For each experiment, the top row presents the (PSNR and SSIM) on the X test set. Due to the simplicity of the dataset, NMF achieved reasonable performance on this dataset. GLOM achieves better SSIM but worse PSNR than NMF while AM performed 1-2dB better. NES achieves much stronger performance than all other methods, achieving about 1dB worse than the fully supervised performance. Initializing NES with the masks obtained by GLOM, in similar performance to the fully-supervised upper bound. FT from AM (numbers for finetuning from AM were omitted from the tables for clarity, as they were inferior to finetuning from GLOM in all experiments) achieved similar performance (24.0/0.95 and 23.8/0.95) to FT from GLOM. In order to evaluate our method on more realistic images, we evaluate on separating mixtures consisting of pairs of images sampled from the Handbags BID33 and Shoes BID31 datasets, which are commonly used for evaluation of conditional image generation methods. To create each Y mixture image, we randomly sample a shoe image from the Shoes dataset and a handbag image from the Handbags dataset and sum them. For the observed distribution, we sample another 5000 different images from a single dataset. We evaluate our method both for cases when the X class is Shoes and when it is Handbags. From the in Tab. 1, we can observe that NMF failed to preserve fine details, penalizing its performance metrics. GLOM (which used a VGG perceptual loss) performed much better, due to greater expressiveness. AM performance was similar to GLOM on this task, as the perceptual loss and stability of training of non-adversarial models helped GLOM greatly. NES performed much better than all other methods, even when initialized from a constant mask. Finetuning from GLOM, helped NES achieve stronger performance, nearly identical to the fully-supervised upper bound. It performed better than finetuning from AM (not shown in table) which achieved 22.5/0.85 and 22.7/0.86. Similar can be drawn from the qualitative comparison in the figure above. Separating environmental noise from speech is a long standing problem in signal processing. Although supervision for both human speech and natural noises can generally be obtained, we use this task as a benchmark to evaluate our method's performance on audio signals where X and B are not dependent. This benchmark is a proxy for tasks for which a clean training set of X sounds cannot be obtained e.g. for animal sounds in the wild, sounds training without animal noises can easily be obtained, but clean sounds made by the animal with no sounds are unlikely to be available. We obtain clean speech segments from the Oxford-BBC Lip Reading in the Wild (LRW) Dataset BID2, and resample the audio to 16 kHz. Audio segments from ESC-50 BID21, a dataset of environmental audio recordings organized into 50 semantic classes, are used as additive noise. Noisy speech clips are created synthetically by first splitting clean speech into clips with duration of 0.5 seconds, and adding a random noise clip, such that the ing SNR is zero. We then compute a mel-scale spectrogram with 64 bins, using STFT with window size of 25 ms, hop length of 10 ms, and FFT size of 512, ing in an input audio feature of 64 × 64 scalars. Finally, power-law compression is performed with p = 0.3, i.e. A 0.3, where A is the input audio feature. From the in Tab. 2, we can observe that GLOM, performed better than Semi-Supervised NMF by about 1dB better. AM training, performed about 2dB better than GLOM. Due to the independence between the sources in this task, NES performed very well, even when trained from a constant mask initialization. Performance was less than 1dB lower than the fully supervised (while not requiring any clean speech samples). In this setting due to the strong performance of NES, initializing NES with the speech estimates obtained by GLOM (or AM), did not yield improved performance. Separating vocal music into singing voice and instrumental music as well as instrumental music and drums has been a standard task for the signal processing community. Here our objective is to understand the behavior of our method in settings where X and B are dependent (which makes synthesis by addition of random X and B training samples a less accurate approximation).For this task we use the MUSDB18 Dataset BID24, which, for each music track, comprises separate signal streams of the mixture, drums, bass, the rest of the accompaniment, and the vocals. We convert the audio tracks to mono, resample to 20480 Hz, and then follow the procedure detailed in Sec. 4.2 to obtain input audio features. From the in Tab. 3, we can observe that NMF was the worst performer in this setting (as its simple bases do not generalize well between songs). GLOM was able to do much better than NMF and was even competitive with NES on Vocal-Instrumental separation. Due to the dependence between the two sources and low SNR, initialization proved important for NES. Constant initialization NES performed similarly to AM and GLOM. Finetuning NES from GLOM masks performed much better than all other methods and was competitive with the supervised baseline. GLOM was much better than AM initialization (not shown in table) that achieved 0.9 and 2.9. GLO vs. Adversarial Masking: GLO Masking as a stand alone technique usually performed worse than Adversarial Masking. On the other hand, finetuning from GLO masks was far better than finetuning from adversarial masks. We speculate that mode collapse, inherent in adversarial training, makes the adversarial masks a lower bound on the X source distribution. GLOM can in models that are too loose (i.e. that also encode samples outside of X). But as an initialization for NES finetuning, it is better to have a model that is too loose than a model which is too tight. Supervision Protocol: Supervision is important for source separation. Completely blind source separation is not well specified and simply using general signal statistics is generally unlikely to yield competitive . Obtaining full supervision by providing a labeled mask for training mixtures is unrealistic but even synthetic supervision in the form of a large training set of clean samples from each source distribution might be unavailable as some sounds are never observed on their own (e.g. sounds of car wheels). Our setting significantly reduces the required supervision to specifying if a certain sound sample contains or does not contain the unobserved source. Such supervision can be quite easily and inexpensively provided. For further sample efficiency increases, we hypothesize that it would be possible to label only a limited set of examples as containing the target sound and not, and to use this seed dataset to finetune a deep sound classifier to extract more examples from an unlabeled dataset. We leave this investigation to future work. To showcase the generality of our method, we chose not to encode task specific constraints. In practical applications of our method however we believe that using signalspecific constraints can increase performance. Examples of such constraints include: repetitiveness of music BID23, sparsity of singing voice, smoothness of natural images. Non-Adversarial Alternatives: The good performance of GLOM vs. AM on the vocals separation task, suggests that non-adversarial generative methods may be superior to adversarial methods for separation. This has also been observed in other mapping tasks e.g. the improved performance of NAM BID8 over DCGAN. A perfect signal separation function is a stable global minimum of NES as i) the synthetic mixtures are equal to real mixtures ii) real mixtures are perfectly separated. In all NES experiments (with constant, AM or GLOM initialization), NES converged after no more than 10 iterations, typically to different local minima. It is empirically evident that NES is not guaranteed to converge to a global minimum (although it converges to good local minima). We defer formal convergence analysis of NES to future work. In this paper we proposed a novel method-Neural Egg Separation-for separating mixtures of observed and unobserved distributions. We showed that careful initialization using GLO Masking improves in challenging cases. Our method achieves much better performance than other methods and was usually competitive with full-supervision. GLOM and AM use the same generator and discriminator architectures respectively for audio as they do for images. They operate on mel-scale spectrogram at 64 × 64 resolution. Masking Network: The generator for AM operates on 64 × 64 mel-scale audio spectrograms. It consists of 3 convolutional and 3 deconvolutional layers with stride 2 and no pooling. Outputs of convolutional layers are normalized with BatchNorm and rectified with ReLU activation, except for the last layer where sigmoid is used. In addition to the LSGAN loss, an additional magnitude loss is used, with relative weight of λ = 1.NES and the supervised method operate on full linear spectrogram of dimensions 257 × 64, without compression. They use the same DiscoGAN architecture, which contains two additional convolutional and deconvolutional layers. In this section we describe our implementation of the NMF semi-supervised source separation baseline BID26. NMF trains a decomposition: B = W Z where W are the weights and Z = [z 1, ..., z N] are the per sample latent codes. Both W and Z are non-negative. Regularization is important for the performance of the method. We follow BID9 BID12 and use L 1 regularization to ensure sparsity of the weights. The optimization problem therefore becomes: We present a qualitative analysis of the of GLOM and NES. To understand the quality of generations of GLO and the effect of the masking function, we present in Fig.2 the of the GLO generations given different mixtures from the Speech dataset. We also show the after the masking operation described in Eq. 10. It can be observed that GLO captures the general features of the sources, but is not able to exactly capture fine detail. The masking operation in GLOM helps it recover more fine-grained details, and in much cleaner separations. DISPLAYFORM0 We also show in Fig.2 the evolution of NES as a function of iteration for the same examples. NES(k) denotes the of NES after k iterations. It can be seen the NES converges quite quickly, and improve further with increasing iterations. In FIG1, we can observe the performance of NES on the Speech dataset in terms of SDR as a function of iteration. The are in line with the qualitative examples presented before, NES converges quickly but makes further gains with increasing iterations.

Represent the TLDR for finding the corresponding scientific article

An iterative neural method for extracting signals that are only observed mixed with other signals

represent the short summary for finding the corresponding scientific article (long)

We learn an unsupervised learning algorithm that produces useful representations from a set of supervised tasks. At test-time, we apply this algorithm to new tasks without any supervision and show performance comparable to a VAE.


Question: Represent the article for finding a single-sentence TLDR

Question: In health, machine learning is increasingly common, yet neural network embedding (representation) learning is arguably under-utilized for physiological signals. This inadequacy stands out in stark contrast to more traditional computer science domains, such as computer vision (CV), and natural language processing (NLP). For physiological signals, learning feature embeddings is a natural solution to data insufficiency caused by patient privacy concerns -- rather than share data, researchers may share informative embedding models (i.e., representation models), which map patient data to an output embedding. Here, we present the PHASE (PHysiologicAl Signal Embeddings) framework, which consists of three components: i) learning neural network embeddings of physiological signals, ii) predicting outcomes based on the learned embedding, and iii) interpreting the prediction by estimating feature attributions in the "stacked" models (i.e., feature embedding model followed by prediction model). PHASE is novel in three ways: 1) To our knowledge, PHASE is the first instance of transferal of neural networks to create physiological signal embeddings. 2) We present a tractable method to obtain feature attributions through stacked models. We prove that our stacked model attributions can approximate Shapley values -- attributions known to have desirable properties -- for arbitrary sets of models. 3) PHASE was extensively tested in a cross-hospital setting including publicly available data. In our experiments, we show that PHASE significantly outperforms alternative embeddings -- such as raw, exponential moving average/variance, and autoencoder -- currently in use. Furthermore, we provide evidence that transferring neural network embedding/representation learners between distinct hospitals still yields performant embeddings and offer recommendations when transference is ineffective. Representation learning (i.e., learning embeddings) BID14 has been applied to medical images and clinical text (; BID16 BID13) but has been under-explored for time series physiological signals in electronic health records. This paper introduces the PHASE (PHysiologicAl Signal Embeddings) framework to learn embeddings of physiological signals FIG1 ), which can be used for various prediction tasks FIG1, and has been extensively tested in terms of its transferability using data from multiple hospitals (FIG1). In addition, this paper introduces an interpretability method to compute per-sample feature attributions of the original features (i.e., not embeddings) for a prediction in a tricky "stacked" model situation (i.e., embedding model followed by prediction model) (FIG1).Based on computer vision (CV) and natural language processing (NLP), exemplars of representation learning, physiological signals are well suited to embeddings. In particular, CV and NLP share two notable traits with physiological signals. The first is consistency. For CV, the domain has consistent features: edges, colors, and other visual attributes. For NLP, the domain is a particular language with semantic relationships consistent across bodies of text. For sequential signals, physiological patterns are arguably consistent across individuals. The second attribute is complexity. Across these three domains, each particular domain is sufficiently complex such that learning embeddings is non-trivial. Together, consistency and complexity suggest that for a particular domain, every research group independently spends a significant time to learn embeddings that may ultimately be Figure 1: The PHASE framework, which consists of embedding learning, prediction, interpretation, and transference. The checkered patterns denote that a model is being trained in the corresponding stage, whereas solid colors denote fixed weights/models. The red side of the LSTM denotes the hidden layer we will use to generate embeddings. In (c), the size of the black circles on the left represent the feature attributions being assigned to the original input features. The signals and the outputs of the LSTMs are vectors. Multiple connections into a single XGB model are simply concatenated. More details on the experimental setup can be found in Sections 4.1 and 6.1.quite similar. In order to avoid this negative externality, NLP and CV have made great progress on standardizing their embeddings; in health, physiological signals are a natural next step. Furthermore, physiological signals have unique properties that make them arguably better suited to representation learning than traditional CV and NLP applications. First, physiological signals are typically generated in the health domain, which is constrained by patient privacy concerns. These concerns make sharing data between hospitals next to impossible; however, sharing models between hospitals is intuitively safer and generally accepted. Second, a key component to successful transfer learning is a community of researchers that work on related problems. According to , there were at least fifty-three research publications using deep learning methods for physiological signals in the past ten years. Additionally, we discuss particular examples of neural networks for physiological signals in Section 2.2. These varied applications of neural networks imply that there is a large community of machine learning research scientists working on physiological signals, a community that could one day work collaboratively to help patients by sharing models. Although embedding learning has many aforementioned advantages, it makes interpretation more difficult. Naive applications of existing interpretation methods (; ; do not work for models trained using learned embeddings, because they will assign attributions to the embeddings. Feature attributions assigned to embeddings will be meaningless, because the embeddings do not map to any particular input feature. Instead, each embedding is a complicated, potentially non-linear combination of the original raw physiological signals. In a health domain, inability to meaningfully interpret your model is unsatisfactory. Healthcare providers and patients alike generally want to know the reasoning behind predictions/diagnoses. Interpretability can enhance both scientific discovery as well as provide credibility to predictive models. In order to provide a principled methodology for mapping embedding attributions back into physiological signal attributions, we provide a proof that justifies PHASE's Shapley value framework in Section 3.3. This framework generalizes across arbitrary stacked models and currently encompasses neural network models (e.g., linear models, neural networks) and tree-based models (e.g., gradient boosting machines and random forests).In the following sections, we discuss previous related work (Section 2) and describe the PHASE framework (Section 3). In Section 4, we first evaluate how well our neural network embeddings make accurate predictions (Section 4.2.1). Second, we evaluate whether transferring these embedding learners still enables accurate predictions across three different hospitals separated by location and across hospital departments (Section 4.2.2). Lastly, we present a visualization of our methodology for providing Shapley value feature attributions through stacked models in Section 4.2.3. Representation learning (embedding learning) in health is growing more popular. One particularly natural subdomain is medical image analysis, e.g., mammography analysis, kidney detection in ultrasound images, optical coherence tomography image analysis, diagnosing pneumonia using chest X-ray images, lung pattern analysis, otitis media image analysis, and more BID1 BID16 BID6 BID8; ). Outside of image analysis, additional examples of transfer learning in the medical domain include BID13, Wiens et al., Brisimi et al. (2018, Choi et al., Choi et al., and Che et al. (2016 . Even within physiological signals, some examples of embedding learning are beginning to sprout up, including , who utilize kNNs to perform transfer learning for brain-computer interaction. Comparatively, PHASE transfers neural networks as embedding functions learned in an partially supervised manner, where the embeddings provide a basis for training a model on any prediction task (as opposed to being tied to the prediction they were trained on). We denote partially supervised networks to be networks trained with prediction tasks related to the final downstream prediction. To our knowledge, our work is the first to transfer deep neural networks for embedding sequential physiological signals, embeddings which are not tied to a particular prediction problem. use acoustic signals to detect anomalies in heart sound. Based on this substantive community of research scientists working on physiological signals, there is a clear opportunity to unify independent research by appropriately using partially supervised feature embedding learning. In the vein of embedding learning, BID14 applied autoencoders to blood volume pulse and skin conductance measured from 36 people playing a video game and used the encodings to predict affective state. In their paper, the sample size is fairly small and reflects that their primary objective was to perform feature extraction and feature selection. In contrast, PHASE evaluates transferring embedding learners (i.e., feature extractors) across multiple hospitals TAB5.2.3 FORECASTING FOR OPERATING ROOM DATA proposed an approach, namely Prescience, which achieved state-of-the-art hypoxemia predictions using operating room data, the same data we used to evaluate PHASE. Prescience utilizes gradient boosting machines (GBM) applied to features extracted by using traditional time series feature extraction methods -exponential moving average/variance embeddings. Prescience compares prediction models including gradient boosting machines, linear lasso, linear SVM, and a parzen window with the objective of forecasting low blood oxygen in the future. Ultimately, find the highest performing method to be GBM trees. With samples drawn from the same data set, PHASE seeks to substitute the feature extraction used in Prescience with a deep learning approach, which ed in a better average precision compared to Prescience without the clinical text features (4 is Prescience and 12 is PHASE in FIG3). Interpretability of models via local feature attributions has been addressed in numerous independent pieces of work (; ; . For our evaluation of interpretability, we choose to focus on Shapley values introduced by Lloyd Shapley, originally in the context of game theory BID18 . identify Shapley values as the only additive feature attribution method that satisfies the properties of local accuracy, missingness, and consistency. For PHASE, our pipeline includes multiple models -GBM trees and LSTM networks. Methods exist for obtaining Shapley values for GBM trees (Tree SHAP) and for neural networks (Deep LIFT/Deep SHAP);. However, the default version of these methodologies do not beget a theoretically justified approach for propagating attributions through multiple models. In fact, to the authors' knowledge, a tractable method for obtaining local feature attributions for mixes of neural networks and trees does not exist. In this paper, we utilize versions of Tree SHAP and Deep LIFT that create single reference attributions that can be composed to address stacked models. At the end of obtaining many attributions, the average of these attributions approximates the Shapley value attributions (more details in Section 3.3). Taking inspiration from sinusoidal waveforms, we name our methodology PHASE. In the PHASE framework, the first step is to learn neural network embeddings for physiological signals (FIG1). The second step is to predict outcomes based on the learned feature embedding (as in FIG1), potentially across multiple hospitals (as in FIG1). Finally, the last step is to interpret the prediction by estimating feature attributions through the models trained in the first two steps (as in FIG1). PHASE uses LSTM networks to learn feature embeddings from time series physiological data. LSTMs are a popular variant on recurrent neural networks introduced by BID3. They have the capacity to model long term dependencies, while avoiding the vanishing gradient problem BID15. Details on the model architecture may be found in Section 6.2.For PHASE, we first train a univariate LSTM on each physiological signal P, predicting the minimum of P in the future five minutes FIG1 ). Note that we choose the minimum of the next five minutes because we care about forecasting adverse outcomes. Then, we obtain hidden embeddings of the original physiological signals by passing them through to the hidden layer (the red layer in FIG1). These embeddings are unsupervised in the sense that training them simply requires the same feature (albeit at different time steps). Yet, they are supervised in that we specify our interest to be forecasting adverse outcomes constituted by low signal values. We choose to focus on the minimum, because adverse outcomes in physiological signals are often tied to too-low signals. We find that the completely unsupervised alternative of an LSTM autoencoder is significantly less performant than the LSTM trained to predict the minimum of the next five minutes (8 is autoencoder and 12 is PHASE in FIG3).One reason behind having univariate neural networks is for transference. By using univariate networks, the input to the final prediction model may be any set of physiological signals with existing embedding learners. This is especially useful because hospital departments have substantial variation in the signals they may choose to collect for features. Another reason for univariate networks is that data in a single hospital is often collected at different points in time, or new measurement devices may be introduced to data collection systems. For traditional pipelines, it may be necessary to re-train entire machine learning pipelines when new features are introduced. With univariate networks, the flexibility would mean pre-existing embedding learners would not necessarily need to be re-trained. PHASE can use any prediction model. In this paper, we focus on gradient boosting machine trees because the Prescience method found that they outperform several other models in the operating room data. Gradient boosting machines were introduced by. This technique creates an ensemble of weak prediction models in order to perform classification/regression tasks in an iterative fashion. In particular, we utilize XGBoost, a popular implementation of gradient boosting machines that uses additive regression trees . XGBoost often dominates in Kaggle, a platform for predictive modeling competitions. In particular, seventeen out of twenty nine challenge winning solutions used XGBoost in 2015 . For PHASE, we postulate that utilizing embeddings of time series signals provides stronger features for the ultimate prediction with XGB (as visualized in FIG1 . Details on the model architecture may be found in Section 6.2. PHASE addresses an inherent challenge in the interpretation of an embedding model (or feature representation model). Estimating feature attributions is a common way to make a prediction interpretable. At a high level, the goal is to explain how much each feature matters for a model's prediction. However, this goal is only meaningful if the model being explained uses features with a natural human interpretation. For example, if we interpret PHASE's GBM model, which takes the embeddings as input and outputs a prediction, our feature attributions will be assigned to the embeddings, which are not meaningful to doctors or patients. The answer is to extend the prediction model (here, a GBM) by combining the feature embedding model (here, a LSTM network), which makes a "stacked" model (FIG1). Since the original features in the embedding stage are meaningful, one solution is to utilize a model agnostic feature attribution method over the "stacked" model. For our attributions, we aim to provide Shapley values, for which the exact model agnostic computation has an exponential computational complexity DISPLAYFORM0 where N is the sample size and M is the number of features) BID18. In response, one might want to use a model-specific method of computing approximate Shapley values to gain speed by knowing the model. However, to the authors' knowledge, there was previously no model-specific method to estimate Shapley values for a stack comprised of LSTMs and a GBM (or even local feature attributions for that matter).Single reference Shapley values: Our new method for estimating Shapley values for the aforementioned stacked model (i.e., LSTMs and GBM), requires adaptations on two existing feature attributions methods. First is Deep SHAP, a variant on Deep LIFT -a feature attribution method for neural networks ). Deep SHAP differs from Deep LIFT in that it can find attributions for single references. Both methods can be written as modifications to a traditional backward pass through a neural network BID0. Since the computational complexity of a backward pass is the same as a forward pass through the network, we can consider this cost "low". The second method we utilize is "Independent Tree SHAP". This method is a variation on normal Tree SHAP, but it can be computed for single references. Independent Tree SHAP has a computational complexity of O(M LT), where L is the maximum number of leaves in any given tree and T is the number of trees in the GBM. "Stacked" model Shapley values: Combining these two methods amounts to treating the "stacked" model (FIG1) as a larger neural network and applying Deep SHAP to pass back attributions as gradients at each layer BID0. However, at the GBM layer we obtain the appropriate gradients by dividing the Independent Tree SHAP Shapley values by the difference between the sample and the references. According to Theorem 1, we can then average over these single reference attributions for an approximation to the Shapley values. Generalizability: Note that Theorem 1 also implies that for any arbitrary set of models in a stack, if single reference Shapley values are obtainable for each model, the Shapley values for the entire stack can be obtained. Because the single reference Shapley value methods are known for neural networks and for trees, any "stacked" model composed of these two methods can be explained. Worth noting is that many embedding/prediction models can be represented as neural networks, making our framework to attribute "stacked" models fairly general. DISPLAYFORM1 where D is the data distribution, F is the set of all features, and f is our model. Rewriting the sum over all permutations of F, rather than over all combinations, the weighting term becomes one: DISPLAYFORM2 where the last step depends on independence between the permutations, independence between the conditional and non-conditional sets, and the data generating mechanism. We first describe our data sets, evaluation metric, model architectures, and the of comparisons between PHASE and alternative approaches in various testing scenarios (Section 4.2). More details on the model architecture and the models used for each experiment can be found in the Appendix. Data Description Hospital 0/1 data was collected via the Anesthesia Information Management System (AIMS), which records all data measured in the operating room during surgery. Both medical centers are within the same city (within 10 miles of each other). Hospital P is a sub-sampled version of the publicly available MIMIC data set from PhysioNet, which contains data obtained from an intensive care unit in Boston, Massachusetts BID5. Hospital P data was collected several thousands of miles from the medical centers associated with hospital 0/1 data. Some details about these hospitals are in Table 1, with more in Appendix: Section 6.1. Table 1: Statistics of the different data sources. Hospital P is a public data set (PhysioNet). Additional details about hospital 0/1 data in Figures 5, 6, and 7 in Appendix: Section 6.1 The hospital 0/1 data includes static information (height, weight, age, sex, procedure codes), as well as real-time measurements of thirty-five physiological signals (e.g., SaO 2, FiO 2, ETCO 2, etc.) sampled minute by minute. Although the hospital P data contains several physiological signals sampled at a high frequency, we solely use a minute by minute SaO 2 signal for our experiments. Any missing values in the data are imputed by the mean and each feature is standardized to have unit mean and variance. One important note is that although hospitals 0 and 1 are spatially close, one is an academic medical center and one is a trauma center. More details on the patients from these hospitals can be found in Section 6.1.Evaluation Methodology PHASE and alternative approaches are evaluated based on real-time prediction tasks, for example, whether a certain condition will occur in the next 5 minutes (details in Sections 4.2 and 6.1). Our evaluation metric for prediction performance in binary classification is area under the precision-recall curve, otherwise known as average precision (AP). Rather than ROC curves, PR curves often better highlight imbalanced labels. Precision is defined as tp tp+f p and recall is tp tp+f n, where tp is true positives, f p false positives, and f n false negatives. The area under the curve provides a summary statistic that balances both precision and recall. In this section, we compare different embeddings of physiological signals as discussed in TAB2 (where Min h represents PHASE). Based on these comparisons, we see the overall performance of predicting three adverse clinical events in Section 4.2.1 as well as a discussion of how well the embedding learners transfer between hospitals in Section 4.2.2. Lastly, in Section 4.2.3, we depict the attributions from our new model stacking method for obtaining Shapley values. In the operating room, there are a few physiological signals that stand out as indicators of adverse outcomes. Three of these signals include SaO 2 (blood oxygen), ETCO 2 (end tidal CO2), and NIBPM (noninvasive blood pressure measurement), which are linked to our three adverse outcomes: hypoxemia, hypocapnia, and hypotension, respectively. Forecasting these outcomes is particularly important because deviations from the norm could spell disaster; ). More details on labels in Section 6.1. Hidden layer embedding from an LSTM trained to predict the minimum of the current signal five minutes into the future on hospital h's data. In this section, we aim to investigate the performance gains PHASE embeddings offer over other embeddings. FIG3 shows the performance of XGB with different representations of the same signals across three prediction tasks. In terms of pre-training for this experiment, there is none for the Raw and EMA embeddings. However for Min h and Auto h, we have trained fifteen univariate LSTM networks for both objectives across both hospitals (a total of sixty networks, not including fine-tuned networks). We fix these same LSTM networks to generate hidden embeddings of the original signals across the three final prediction tasks of hypoxemia, hypocapnia, and hypotension. In terms of performance, the average precision points with all signals (in blue) are almost always significantly above their associated average precision points using only a single signal (in red). This suggests the outcomes derived from forecasting a single signal are complex and benefit from having access to more signals. Most importantly, with the exception of the fine tuned model, the Min h (PHASE) models (10 and 12) consistently outperform all other models in FIG3 by a significant margin. The fact that LSTM autoencoders fail to grasp meaningful representations of the data in comparison to partially supervised LSTMs offers an insight; on our data set, the key to performant embeddings is to have closeness between the LSTM's prediction task and the downstream prediction task. In this case, effective embeddings for forecasting adverse outcomes related to too-low signals, required that the embeddings themselves were related to low signals in the future. In this section, we have two aims: the first aim is to evaluate how well our embedding models transfer. The second aim is to explore methods to repurpose them when there is a large amount of domain shift between hospitals. First, we can look back to FIG3. The feature embeddings learned in a source hospital that differs to the target hospital performs significantly better than the EMA (Prescience -) and Raw embeddings (2 and 4) and generally on par with a matching source and target hospital. This is promising, because it suggests that the domain shift between hospitals 0 and 1 does not prevent physiological signal embeddings from transferring well. Although these hospitals do have similar patient distributions, this transference is better than TAB14 might be expected, given that one hospital is an academic medical center and the other is a trauma center. As further evidence of their differences, we report the top ten diagnoses from each hospital in Appendix: Section 6.1 and find no overlap apart from CALCULUS OF KIDNEY between hospitals 0 and 1.Next, in Figure 3, we can see that the Min P embeddings created from the publicly available PhysioNet data (2 p) are worse representations of SaO 2 in comparison to the embeddings trained with the target hospital's data. This implies that the domain shift between hospital P and the OR hospitals is too large -ing in learned Min embeddings (2 P) that are not as useful for prediction. Next, the improvement of the fine tuned models (4 P) over Min P embeddings, suggests the following insight: that fine tuning serves to recover the performance lost due to transference across distributionally disparate hospitals. Then, observing the improvement of Hypox P embeddings (6 P) over Min P embeddings, conveys a natural insight: that closeness between LSTM prediction tasks and the downstream prediction task is beneficial in the face of transference. This naturally extends the observation from Section 4.2.1: that closeness between LSTM prediction tasks and the downstream prediction task is beneficial in the face of performance. These two observations suggest two approaches to transferring across substantially different hospital data sets: 1.) train LSTMs with very specific prediction tasks that match the downstream prediction and 2.) fine tune LSTM networks. In this section, our aim is to evaluate the efficacy of our interpretability method for stacked models. To do so, we separate interpretability evaluation approaches into two categories: qualitative and quantitative. Qualitative evaluations are important to ensure human insight into interpretability methods. However, our primary goal is to ensure that our novel method to obtain local feature attributions for stacked models is correct in the sense that the attributions estimate Shapley values. One qualitative evaluation of feature attributions is who demonstrate that local feature attributions improve the performance of practicing anesthesiologists in forecasting hypoxemia. Our quantitative validation is a standard ablation/perturbation test, in a similar fashion to other interpretability evaluations: BID2, BID4, BID0, and. The test consists of the following. For a single sample, we sort the input features according to their attributions, and iteratively impute each feature by the mean of the last two minutes. In order to ensure our interpretations generalize, we evaluate on the test set. Additionally, we use the top 1000 positive samples sorted by the predicted probability of hypoxemia (true positives). Then we evaluate the mean predicted probability across all samples, which will start high (for true positives) and monotonically decrease as we impute features, leading to an overall decrease in the average probability. Good interpretability methods should in an initial steepness because the most "important" hypoxemic pathology is imputed first. Figure 4: Ablation test on the top 1000 positive labels, sorted by the probability prediction of the final model. We "remove" the features (by imputing the mean of the last two minutes) according to Shapley values or a random ordering and then predict the probability of hypoxemia on the entirety of our test set. We obtain Shapley values for both models with a fixed set of 100 samples. The model stack used for the interpretability evaluation (right) is in FIG3. More details about the setup for this experiment in Section 6.5Deep SHAP was originally proposed for traditional deep learning models, we extend it to support a mixture of model types in PHASE. As such, our primary aim is to evaluate against the pre-existing Deep SHAP methodology. We compare LSTM embeddings fed into XGB (PHASE) against LSTM embeddings fed into an MLP, because original Deep SHAP supports only traditional neural network architecture (see FIG1 in the Appendix for more details on the model setups). In Figure 4, we verify that ordering feature imputation by our interpretability method which combines Deep SHAP with Independent Tree SHAP (LSTM→XGB) does lead to initial steepness of the predicted probability of hypoxemia in a similar fashion to Deep SHAP alone (LSTM→MLP), with both methods outperforming a random ordering. In fact, it appears that for the Figure 4 (right), imputing our attributions for LSTM→XGB has a potentially more destructive effect on performance in the early number of imputed features. Lastly, in Appendix Section 6.5.1, we show attributions from both methods in a informal visual evaluation to confirm concordance. This paper presents PHASE, a new approach to machine learning with physiological signals based on transferring embedding learners. PHASE has potentially far-reaching impacts, because neural networks inherently create an embedding before the final output layer. As discussed in Section 2.2, there is a large body of research independently working on neural networks for physiological signals. PHASE offers a potential method of collaboration by analyzing partially supervised univariate networks as semi-private ways to share meaningful signals without sharing data sets. In the section we offer several insights into transference of univariate LSTM embedding functions. First, closeness of upstream (LSTM) and downstream prediction tasks is indeed important for both predictive performance and transference. For performance, we found that predicting the minimum of the future five minutes was sufficient for the LSTMs to generate good embeddings. For transference, predicting the minimum of the next five minutes was sufficient to transfer across similar domains (operating room data from an academic medical center and a trauma center) when predicting hypoxemia. However when attempting to utilize a representation from Hospital P, we found that the difference between operating rooms and intensive care units was likely too large to provide good predictions. Two solutions to this include fine tuning the Min LSTM models as well as acknowledging the large amount of domain shift and training specific LSTM embedding models with a particular downstream prediction in mind. Last but not least, this paper introduced a way to obtain feature attributions for stacked models of neural networks and trees. By showing that Shapley values may be computed as the mean over single reference Shapley values, this model stacking framework generalizes to all models for which single reference Shapley values can be obtained, which was quantitatively verified in Section 4.2.3.We intend to release code pertinent to training the LSTM models, obtaining embeddings, predicting with XGB models, and model stacking feature attributions -submitted as a pull request to the SHAP github (https://github.com/slundberg/shap). Additionally, we intend to release our embedding models, which we primarily recommend for use in forecasting "hypo" predictions. In the direction of future work, it is important to carefully consider representation learning in health -particularly in light of model inversion attacks as discussed in. To this end, future work in making precise statements about the privacy of models deserves attention, for which one potential avenue may be differential privacy . Other important areas to explore include extending these to higher sampling frequencies. Our data was sampled once per minute, but higher resolution data may beget different neural network architectures. Lastly, further work may include quantifying the relationship between domain shifts in hospitals and PHASE and determining other relevant prediction tasks for which embeddings can be applied (e.g., "hyper" predictions, doctor action prediction, etc. Labels For hypoxemia, a particular time point t is labelled to be one if the minimum of the next five minutes is hypoxemic (min(SaO t+1:t+6 2) ≤ 92). All points where the current time step is currently hypoxemic are ignored (SaO t 2 ≤ 92). Additionally we ignore time points where the past ten minutes were all missing or the future five minutes were all missing. Hypocapnia and hypotension are only labelled for hospitals 0 and 1. Additionally, we have stricter label conditions. We labeled the current time point t to be one if (min(S t−10:t) > T ) and the minimum of the next five minutes is "hypo" (min(S t+1:t+5) ≤ T ). We labeled the current time point t to be zero if (min(S t−10:t) > T ) and the minimum of the next ten minutes is not "hypo" (min(S t+1:t+10) > T ). All other time points were not considered. For hypocapnia, the threshold T = 34 and the signal S is ETCO 2. For hypotension the threshold is T = 59 and the signal S is NIBPM. Additionally we ignore time points where the past ten minutes were all missing or the future five minutes were all missing. As a , we have different sample sizes for different prediction tasks (reported in TAB7). For Min predictions, the label is the value of min(S t+1:t+5), points without signal for in the future five minutes are ignored. For Auto predictions, the label is all the time points: S t−59:t. The sample sizes for Min and Auto are the same and are reported in Table 3. Table 3: Sample sizes for the Min and Auto predictions for training the LSTM autoencoders. For the autoencoders we utilize the same data, without looking at the labels. We only utilize the 15 features above the line in both hospitals (Figure 5) for training our models., implemented in the Keras library with a Tensorflow back-end. We train our networks with either regression (Auto and Min embeddings) or classification (Hypox) objectives. For regression, we optimize using Adam with an MSE loss function. For classification we optimize using RMSProp with a binary cross-entropy loss function (additionally, we upsample to maintain balanced batches during training). Our model architectures consist of two hidden layers, each with 200 LSTM cells with dense connections between all layers. We found that important steps in training LSTM networks for our data are to impute missing values by the training mean, standardize data, and to randomize sample ordering prior to training (allowing us to sample data points in order without replacement). To prevent overfitting, we utilized dropouts between layers as well as recurrent dropouts for the LSTM nodes. Using a learning rate of 0.001 gave us the best final . The LSTM models were run to convergence (until their validation accuracy did not improve for five rounds of batch stochastic gradient descent). In order to train these models, we utilize three GPUs (GeForce GTX 1080 Ti graphics cards). DISPLAYFORM0 We train GBM trees in Python using XGBoost, an open source library for gradient boosting trees. XGBoost works well in practice in part due to it's ease of use and flexibility. Imputing and standardizing are unnecessary because GBM trees are based on splits in the training data, implying that scale does not matter and missing data is informative as is. We found that a learning rate of 0.02 for hypoxemia (0.1 for hypotension and hypocapnia), a max tree depth of 6, subsampling rate of 0.5, and a logistic objective gave us good performance. All XGB models were run until their validation accuracy was non-improving for five rounds of adding estimators (trees). In order to train these models, we utilize 72 CPUs (Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz) FIG3, apart from the fine tuned LSTMs, which were trained identically but were initialized with the non-target hospital's corresponding LSTM. This figure is for hypoxemia, but hypocapnia and hypotension parallel this setup. Not counting fine tuned LSTMs, there are a total of 60 LSTMs: 30 Auto models for hospitals 0/1 and 30 Min models for hospitals 0/1. LSTM/XGB architecture and hyperparameters are consistent across models and can be found in Section 6.2. The signals and the outputs of the LSTMs are vectors. Multiple connections into a single model are simply concatenated. For all LSTMs, they consist of two layers each with 200 LSTM cells, trained in identical manners, as described in Section 6.2. For XGB, the training is detailed in Section 6.2 as well. The univariate predictions made in FIG3 are similarly obtained, but only utilize the single feature used to obtain the final prediction. Here, "Hypoxemia" means: "Is min(SaO2 (t+1,···,t+5) ) ≤ 92?". EMA MinAuto FIG3. Reporting adjusted p-values based on one hundred bootstraps of the test set with adjusted pairwise comparisons via ANOVA with Tukey's HSD test. 0.0 denotes a p-value less than 1e−14. Since we are most concerned with models 10, 12, and 14 we only report pairs that include these models for the sake of brevity. FIG3. Reporting adjusted p-values based on one hundred bootstraps of the test set with adjusted pairwise comparisons via ANOVA with Tukey's HSD test. 0.0 denotes a p-value less than 1e−14. Since we are most concerned with models 10, 12, and 14 we only report pairs that include these models for the sake of brevity. Leave One Signal Out Test: In FIG12, we create a simulated setting -when predicting each event, we excluded the corresponding physiological signal from our features. For example, we assumed that SaO 2 is not recorded when predicting hypoxemia. Under this setting, we must rely on the remaining signals to predict hypoxemia. This setting is a more unsupervised evaluation in the sense that our outcome is not derived from a signal we create an embedding for. As our show FIG12 ), PHASEs outperformance is consistent for hypocapnia and hypotension. For hypoxemia, all representations perform poorly because predicting hypoxemia heavily relies on SaO2, leaving little signal for the remaining features. This is likely due in part to the low base rates of hypoxemia: 1.06% in hospital 0 and 2.32% in hospital 1. Investigating further, we found that the log loss of the Min embeddings was lower than other embeddings on the validation set, but not on the test set. This overfitting further suggests that there was little signal to be captured, causing simpler embeddings like EMA to be favored. Figure 3. Showcasing what models are being used for the the transference experiment in Figure 3. LSTM/XGB architecture and hyperparameters are consistent across models and can be found in Section 6.2. The signals and the outputs of the LSTMs are vectors. Multiple connections into a single XGB model are simply concatenated. All LSTMs consist of two layers each with 200 LSTM cells, trained in identical manners, as described in Section 6.2. For XGB, the training is detailed in Section 6.2 as well. The univariate predictions made in Figure 3 are similarly obtained, but only utilize the single feature used to obtain the final prediction. Here, "Hypoxemia" means: "Is min FIG1 ) ≤ 92?". 9e−09 7e−13 1 P 7e−13 7e−13 7e−13 3 P 3e−10 7e−13 1.000 7e−13 5 P 1e−07 7e−13 1.000 7e−13 0.998 2 0.736 7e−13 8e−13 7e−13 8e−13 1e−12 4 1e−12 7e−13 7e−13 7e−13 7e−13 7e−13 8e−08 6 7e−13 7e−13 7e−13 7e−13 8e−13 7e−13 7e−13 7e−13 2 P 7e−13 7e−13 7e−13 4e−12 7e−13 7e−13 8e−13 0.002 7e−13 4 P 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 0.979 7e−13 6 P 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 0.333 7e−13 0.988 1e−10 8e−13 1 P 7e−13 0.981 4e−11 3 P 7e−06 7e−13 7e−13 7e−13 5 P 0.229 7e−13 8e−13 7e−13 0.260 2 0.780 7e−13 8e−13 7e−13 0.027 1.000 4 9e−12 7e−13 7e−13 7e−13 0.574 1e−04 1e−06 6 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 2 P 0.603 7e−13 8e−13 7e−13 0.061 1.000 1.000 6e−06 7e−13 4 P 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 1e−04 7e−13 6 P 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 7e−13 4e−08 7e−13 0.953 6.5 INTERPRETABILITY FIG1: Model setup for Figure 4. Showcasing what models are being used to evaluate interpretability. LSTM/XGB architecture and hyperparameters are consistent across models and can be found in Section 6.2. The signals and the outputs of the LSTMs are vectors. Multiple connections into a single XGB model are simply concatenated. Here, "Hypoxemia" means: "Is min FIG1 ) ≤ 92?". Of special note, the MLP is trained identically to the Hypox models. The architecture is a single layer with 100 nodes with a relu activation connected densely into a sigmoid output node. The MLP is trained until convergence by upsampling the number of positive samples to match the negative samples for each batch. The attributions for LSTM→MLP are computed via Deep SHAP and the attributions for LSTM→XGB are computed via Deep SHAP combined with Independent Tree SHAP (our novel method). Both methods use a fixed set of 100 randomly sampled points from the test set. FIG1, we present the local feature attributions for two different model stacks across a random set of examples. We can see that the models generally appear to agree on their predictions, although there are occasional disagreements -which are likely due to the fact that these attributions are for two different models. Being able to observe these trends is useful to understanding models and to achieving credibility. As our primary aim is to ensure that our model stacking local feature attributions agree with feature attributions on neural networks, we also provide attributions for true positives and true negatives in settings where the models agree. DISPLAYFORM0 In FIG1, we present local feature attributions for two different model stacks across true positive examples. Looking at these true positive examples we can see two consistent trends: high variability and a low absolute value of blood oxygen. Looking at the attributions we can discover that the dips in blood oxygen -minute to minute variability was important in both model stacks. Additionally, the closer the time point is to the actual prediction, the more important it is. In FIG1, we present the local feature attributions for two different model stacks across true negative examples. Here we can see that variability and dips make a much smaller relative impact in the model predictions. Instead, the most important factor in determining hypoxemia is the high value of SaO 2 closest to the final prediction. Furthermore, the feature attributions reveal interesting trends in the attributions for the MLP model, where there appears to be a consistent trend even though the samples look fairly different. FIG1: Randomly sampled feature attributions. Local feature attribution plots for two stacked models: 1. LSTM→MLP and 2. LSTM→XGB. Here we present fifteen randomly sampled hypoxemia examples. Figure 13: True negative feature attributions. Local feature attribution plots for two stacked models: 1. LSTM→MLP and 2. LSTM→XGB. Here we present the nine "least probable" positively labelled hypoxemia examples. In order to obtain this set, we took the intersection of the top 1000 negatively labelled examples from both models to get a set of 97 samples and randomly sample nine samples. FIG1: True positive feature attributions. Local feature attribution plots for two stacked models: 1. LSTM→MLP and 2. LSTM→XGB. Here we present the nine "most probable" positively labelled hypoxemia examples. In order to obtain this set, we took the intersection of the top 100 positively labelled examples from both models to get a set of 40 samples and randomly sample nine samples.

Physiological signal embeddings for prediction performance and hospital transference with a general Shapley value interpretability method for stacked models.

Represent the short summary for finding the corresponding scientific article (long)

We can identify prototypical and outlier examples in machine learning that are quantifiably very different, and make use of them to improve many aspects of neural networks.