Question: Represent the scientific article to find a short TLDR

Question: Deep networks often perform well on the data distribution on which they are trained, yet give incorrect (and often very confident) answers when evaluated on points from off of the training distribution. This is exemplified by the adversarial examples phenomenon but can also be seen in terms of model generalization and domain shift. Ideally, a model would assign lower confidence to points unlike those from the training distribution. We propose a regularizer which addresses this issue by training with interpolated hidden states and encouraging the classifier to be less confident at these points. Because the hidden states are learned, this has an important effect of encouraging the hidden states for a class to be concentrated in such a way so that interpolations within the same class or between two different classes do not intersect with the real data points from other classes. This has a major advantage in that it avoids the underfitting which can from interpolating in the input space. We prove that the exact condition for this problem of underfitting to be avoided by Manifold Mixup is that the dimensionality of the hidden states exceeds the number of classes, which is often the case in practice. Additionally, this concentration can be seen as making the features in earlier layers more discriminative. We show that despite requiring no significant additional computation, Manifold Mixup achieves large improvements over strong baselines in supervised learning, robustness to single-step adversarial attacks, semi-supervised learning, and Negative Log-Likelihood on held out samples. Machine learning systems have been enormously successful in domains such as vision, speech, and language and are now widely used both in research and industry. Modern machine learning systems typically only perform well when evaluated on the same distribution that they were trained on. However machine learning systems are increasingly being deployed in settings where the environment is noisy, subject to domain shifts, or even adversarial attacks. In many cases, deep neural networks which perform extremely well when evaluated on points on the data manifold give incorrect answers when evaluated on points off the training distribution, and with strikingly high confidence. This manifests itself in several failure cases for deep learning. One is the problem of adversarial examples , in which deep neural networks with nearly perfect test accuracy can produce incorrect classifications with very high confidence when evaluated on data points with small (imperceptible to human vision) adversarial perturbations. These adversarial examples could present serious security risks for machine learning systems. Another failure case involves the training and testing distributions differing significantly. With deep neural networks, this can often in dramatically reduced performance. To address these problems, our Manifold Mixup approach builds on following assumptions and motivations: we adopt the manifold hypothesis, that is, data is concentrated near a lower-dimensional non-linear manifold (this is the only required assumption on the data generating distribution for Manifold Mixup to work); a neural net can learn to transform the data non-linearly so that the transformed data distribution now lies on a nearly flat manifold; as a consequence, linear interpolations between examples in the hidden space also correspond to valid data points, thus providing novel training examples. (a,b,c) shows the decision boundary on the 2d spirals dataset trained with a baseline model (a fully connected neural network with nine layers where middle layer is a 2D bottleneck layer), Input Mixup with α = 1.0, and Manifold Mixup applied only to the 2D bottleneck layer. As seen in (b), Input Mixup can suffer from underfitting since the interpolations between two samples may intersect with a real sample. Whereas Manifold Mixup (c), fits the training data perfectly (more intuitive example of how Manifold Mixup avoids underfitting is given in Appendix H). The bottom row (d,e,f) shows the hidden states for the baseline, Input Mixup, and manifold mixup respectively. Manifold Mixup concentrates the labeled points from each class to a very tight region, as predicted by our theory (Section 3) and assigns lower confidence classifications to broad regions in the hidden space. The black points in the bottom row are the hidden states of the points sampled uniformly in x-space and it can be seen that manifold mixup does a better job of giving low confidence to these points. Additional in Figure 6 of Appendix B show that the way Manifold Mixup changes the representations is not accomplished by other well-studied regularizers (weight decay, dropout, batch normalization, and adding noise to the hidden states).Manifold Mixup performs training on the convex combinations of the hidden state representations of data samples. Previous work, including the study of analogies through word embeddings (e.g. king -man + woman ≈ queen), has shown that such linear interpolation between hidden states is an effective way of combining factors . Combining such factors in the higher level representations has the advantage that it is typically lower dimensional, so a simple procedure like linear interpolation between pairs of data points explores more of the space and with more of the points having meaningful semantics. When we combine the hidden representations of training examples, we also perform the same linear interpolation in the labels (seen as one-hot vectors or categorical distributions), producing new soft targets for the mixed examples. In practice, deep networks often learn representations such that there are few strong constraints on how the states can be distributed in the hidden space, because of which the states can be widely distributed through the space, (as seen in FIG0). As well as, nearly all points in hidden space correspond to high confidence classifications even if they correspond to off-the-training distribution samples (seen as black points in FIG0). In contrast, the consequence of our Manifold Mixup approach is that the hidden states from real examples of a particular class are concentrated in local regions and the majority of the hidden space corresponds to lower confidence classifications. This concentration of the hidden states of the examples of a particular class into a local regions enables learning more discriminative features. A low-dimensional example of this can be seen in FIG0 and a more detailed analytical discussion for what "concentrating into local regions" means is in Section 3.Our method provides the following contributions:• The introduction of a novel regularizer which outperforms competitive alternatives such as Cutout BID4, Mixup , AdaMix BID10 Dropout . On CIFAR-10, this includes a 50% reduction in test Negative Log-Likelihood (NLL) from 0.1945 to 0.0957.• Manifold Mixup achieves significant robustness to single step adversarial attacks.• A new method for semi-supervised learning which uses a Manifold Mixup based consistency loss. This method reduces error relative to Virtual Adversarial Training (VAT) (a) by 21.86% on CIFAR-10, and unlike VAT does not involve any additional significant computation.• An analysis of Manifold Mixup and exact sufficient conditions for Manifold Mixup to achieve consistent interpolations. Unlike Input Mixup, this doesn't require strong assumptions about the data distribution (see the failure case of Input Mixup in FIG0): only that the number of hidden units exceeds the number of classes, which is easily satisfied in many applications. The Manifold Mixup algorithm consists of selecting a random layer (from a set of eligible layers including the input layer) k. We then process the batch without any mixup until reaching that layer, and we perform mixup at that hidden layer, and then continue processing the network starting from the mixed hidden state, changing the target vector according to the mixup interpolation. More formally, we can redefine our neural network function y = f (x) in terms of k: DISPLAYFORM0 Here g k is a function which runs a neural network from the input hidden state k to the output y, and h k is a function which computes the k-th hidden layer activation from the input x. For the linear interpolation between factors, we define a variable λ and we sample from p(λ). Following , we always use a beta distribution p(λ) = Beta(α, α). With α = 1.0, this is equivalent to sampling from U.We consider interpolation in the set of layers S k and minimize the expected Manifold Mixup loss. DISPLAYFORM1 We backpropagate gradients through the entire computational graph, including to layers before the mixup process is applied (Section 5.1 and appendix Section B explore this issue directly). In the case where k = 0 is the input layer and S k = 0, Manifold Mixup reduces to the mixup algorithm of. With α = 2.0, about 5% of the time λ is within 5% of 0 or 1, which essentially means that an ordinary example is presented. In the more general case, we can optimize the expectation in the Manifold Mixup objective by sampling a different layer to perform mixup in on each update. We could also select a new random layer as well as a new lambda for each example in the minibatch. In theory this should reduce the variance in the updates introduced by these random variables. However in practice we found that this didn't have a significant effect on the , so we decided to sample a single lambda and a randomly chosen layer per minibatch. In comparison to Input Mixup, the in the FIG3 demonstrate that Manifold Mixup reduces the loss calculated along hidden interpolations significantly better than Input Mixup, without significantly changing the loss calculated along visible space interpolations. Our goal is to show that if one does mixup in a sufficiently deep hidden layer in a deep network, then a mixup loss of zero can be achieved so long the dimensionality of that hidden layer dim (H) is greater than the number of classes d. More specifically the ing representations for that class must fall onto a subspace of dimension dim (H) − d. Assume X and H to denote the input and representation spaces, respectively. We denote the labelset by Y and let Z X × Y. Also, let us denote the set of all probability measures on Z by M (Z). Assume G ⊆ H X to be the set of all possible functions that can be generated by the neural network mapping input to the representation space. In this regard, each g ∈ G represents a mapping from input to the representation units. A similar definition can be made for F ⊆ Y H, as the space of all possible functions from the representation space to the output. We are interested in the solution of the following problem, at least in some specific asymptotic regimes: DISPLAYFORM0 where We analyze the above-mentioned minimization when the probability measure P = P D is chosen as the empirical distribution over a finite dataset of size n, denoted by DISPLAYFORM1 DISPLAYFORM2. Let f * ∈ F and g * ∈ G be the minimizers in with P = P D.In particular, we are interested in the case where G = H X, F = Y H, and H is a vector space; These conditions simply state that the two respective neural networks which map input into representation space, and representation space to the output are being extended asymptotically 1. In this regard, we show that the minimizer f * is a linear function from H to Y. This way, it is easy to show that the following equality holds: DISPLAYFORM3 where Proof. With basic linear algebra, one can confirm that the following argument is true as long as DISPLAYFORM4 DISPLAYFORM5 where I d×d and 1 d are the d-dimensional identity matrix and all-one vector, respectively. In fact, b1 T d is a rank-one matrix, while the rank of identity matrix is d. Therefore, A T H only needs to be rank d − 1. DISPLAYFORM6 where h i here means the ith column of matrix H, and ζ i ∈ {1, . . ., d} is the class-index of the ith sample. We show that such selections will make the objective in equal to zero (which is the minimum possible value). More precisely, the following relations hold: DISPLAYFORM7 1 Due to the consistency theorem that proves neural networks with nonlinear activation functions are dense in the function spaceThe final equality is a direct of A T h ζi + b = y ζi for i = 1,..., n. Also, it can be shown that as long as dim (H) > d − 1, then data points in the representation space H have some degrees of freedom to move independently. Corollary 1. Consider the setting in Theorem 1, and assume dim (H) > d − 1. Let g * ∈ G to be the true minimizer of for a given dataset D. Then, data-points in the representation space, i.e. DISPLAYFORM8 Proof. In the proof of Theorem 1, we have DISPLAYFORM9 The r.h.s. of FORMULA11 can become a rank-(d − 1) matrix as long as vector b is chosen properly. Thus, A is free to have a null-space of dimension dim (H)−d+1. This way, one can assign g * (X i) = h ζi +e i, where h j and ζ i (for j = 1, . . ., d and i = 1, . . ., n) are defined in the same way as in Theorem 1, and e i s can are arbitrary vectors in the null-space of A, i.e. e i ∈ ker (A) for all i. This implies that if the Manifold Mixup loss is minimized, then the representation for each class will lie on a subspace of dimension dim (H)−d+1. In the most extreme case where dim (H) = d − 1, each hidden state from the same class will be driven to a single point, so the change in the hidden states following any direction on the class-conditional manifold will be zero. In the more general case with a larger dim (H), a majority of directions in H-space will not change as we move along the class-conditional manifold. Why are these properties desirable? First, it can be seen as a flattening 2. of the class-conditional manifold which encourages learning effective representations earlier in the network. Second, it means that the region in hidden space occupied by data points from the true manifold has nearly zero measure. So a randomly sampled hidden state within the convex hull spanned by the data is more likely to have a classification score that is not fully confident (non-zero entropy). Thus it encourages the network to learn discriminative features in all layers of the network and to also assign low-confidence classification decisions to broad regions in the hidden space (this can be seen in FIG0 and Figure 6). Regularization is a major area of research in machine learning. Manifold Mixup closely builds on two threads of research. The first is the idea of linearly interpolating between different randomly drawn examples and similarly interpolating the labels . These methods encourage the output of the entire network to change linearly between two randomly drawn training samples, which can in underfitting. In contrast, for a particular layer at which mixing is done, Manifold Mixup allows lower layers to learn more concentrated features in such a way that it makes it easier for the output of the upper layers to change linearly between hidden states of two random samples, achieving better (section 5.1 and Appendix B).Another line of research closely related to Manifold Mixup involves regularizing deep networks by perturbing the hidden states of the network. These methods include dropout , batch normalization , and the information bottleneck BID0. and both demonstrated that regularizers already demonstrated to work well in the input space (salt and pepper noise and input normalization respectively) could also be adapted to improve when applied to the hidden layers of a deep network. We believe that the regularization effect of Manifold Mixup would be complementary to that of these algorithms. explored improving adversarial robustness by classifying points using a function of the nearest neighbors in a fixed feature space. This involved applying mixup between each set of nearest neighbor examples in that feature space. The similarity between and Table 1: Supervised Classification Results on CIFAR-10 (a) and CIFAR-100 (b). We note significant improvement with Manifold Mixup especially in terms of Negative log-likelihood (NLL). Please refer to Appendix C for details on the implementation of Manifold Mixup and Manifold Mixup All layers and on SVHN. † and ‡ refer to the reported in and BID10 DISPLAYFORM0 Manifold Mixup is that both consider linear interpolations in hidden states with the same interpolation applied to the labels. However an important difference is that Manifold Mixup backpropagates gradients through the earlier parts of the network (the layers before where mixup is applied) unlike. As discussed in Section 5.1 and Appendix B this was found to significantly change the learning process. AdaMix BID8 is another related method which attempted to learn better mixing distributions to avoid overlap. AdaMix reported 3.52% error on CIFAR-10 and 20.97% error on CIFAR-100. We report 2.38% error on CIFAR-10 and 20.39% error on CIFAR-100. AdaMix only interpolated in the input space, and they report that their method hurt significantly when they tried to apply it to the hidden layers. Thus this method likely works for different reasons from Manifold Mixup and might be complementary. AgrLearn BID9 ) is a method which adds a new information bottleneck layer to the end of deep neural networks. This achieved substantial improvements, and was used together with Input Mixup to achieve 2.45% test error on CIFAR-10. As their method was complimentary with Input Mixup, it's possible that their method is also complimentary with Manifold Mixup, and this could be an interesting area for future work. We present on Manifold Mixup based regularization of networks using the PreActResNet architecture BID11. We closely followed the procedure of as a way of providing direct comparisons with the Input Mixup algorithm. We used weight decay of 0.0001 and trained with SGD with momentum and multiplied the learning rate by 0.1 at regularly scheduled epochs. These for CIFAR-10 and CIFAR-100 are in Table 1a and 1b. We also ran experiments where we took PreActResNet34 models trained on the normal CIFAR-100 data and evaluated them on test sets with artificial deformations (shearing, rotation, and zooming) and showed that Manifold Mixup demonstrated significant improvements (Appendix C Table 5), which suggests that Manifold Mixup performs better on the variations in the input space not seen during the training. We also show that the number of epochs needed to reach good is not significantly affected by using Manifold Mixup in FIG8.To better understand why the method works, we performed an experiment where we trained with Manifold Mixup but blocked gradients immediately after the layer where we perform mixup. On CIFAR-10 PreActResNet18, this caused us to achieve 4.86% test error when trained on 400 epochs and 4.33% test error when trained on 1200 epochs. This is better than the baseline, but worse than Manifold Mixup or Input Mixup in both cases. Because we randomly select the layer to mix, each layer of the network is still being trained, although not on every update. This demonstrates that the Manifold Mixup method improves by changing the layers both before and after the mixup operation is applied. We also compared Manifold Mixup against other strong regularizers. We selected the best performing hyperparameters for each of the following models using a validation set. Using each model's best performing hyperparameters, test error averages and standard deviations for five trials (in %) for CIFAR-10 using PreResNet50 trained for 600 epochs are: vanilla PreResNet50 (4.96 ± 0.19), Dropout (5.09 ± 0.09), Cutout BID4 ) (4.77 ± 0.38), Mixup (4.25 ± 0.11) and Manifold Mixup (3.77 ± 0.18). This clearly shows that Manifold Mixup has strong regularizing effects. (Note that the in Table 1 were run for 1200 epochs and thus these are not directly comparable.)We also evaluate the quality of the representations learned by Manifold Mixup by applying K-Nearest Neighbour classifier on the feature extracted from the top layer of PreResNet18 for CIFAR-10. We achieved test errors of 6.09% (Vanilla PreResNet18), 5.54% (Mixup) and 5.16% (Manifold Mixup). It suggests that Manifold Mixup helps learning better representations. Further analysis of how Manifold Mixup changes the representations is given in Appendix BThere are a couple of important questions to ask: how sensitive is the performance of Manifold Mixup with respect to the hyperparameter α and in which layers the mixing should be performed. We found that Manifold Mixup works well for a wide range of α values. Please refer to Appendix J for more details. Furthermore, the in Appendix K suggests that mixing should not be performed in the layers very close to the output layer. Semi-supervised learning is concerned with building models which can take advantage of both labeled and unlabeled data. It is particularly useful in domains where obtaining labels is challenging, but unlabeled data is plentiful. The Manifold Mixup approach to semisupervised learning is closely related to the consistency regularization approach reviewed by. It involves minimizing loss on labelled samples as well as unlabeled samples by controlling the tradeoff between these two losses via a consistency coefficient. In the Manifold Mixup approach for semi-supervised learning, the loss from labeled examples is computed as normal. For computing loss from unlabelled samples, the model's predictions are evaluated on a random batch of unlabeled data points. Then the normal manifold mixup procedure is used, but the targets to be mixed are the soft target outputs from the classifier. The detailed algorithm for both Manifold Mixup and Input Mixup with semi-supervised learning are given in appendix D.Oliver et al. FORMULA2 performed a systematic study of semi-supervised algorithms using a fixed wide resnet architecture "WRN-28-2" . We evaluate Manifold Mixup using this same setup and achieve improvements for CIFAR-10 over the previously best performing algorithm, Virtual Adversarial Training (VAT) (a) and Mean-Teachers . For SVHN, Manifold Mixup is competitive with VAT and Mean-Teachers. See TAB1. While VAT requires an additional calculation of the gradient and Mean-Teachers requires repeated model parameters averaging, Manifold Mixup requires no additional (non-trivial) computation. In addition, we also explore the regularization ability of Manifold Mixup in a fully-supervised lowdata regime by training a PreResnet-152 model on 4000 labeled images from CIFAR-10. We obtained 13.64 % test error which is comparable with the fully-supervised regularized baseline according to reported in. Interestingly, we do not use a combination of two powerful regularizers ("Shake-Shake" and "Cut-out") and the more complex ResNext architecture as in Oliver et al. FORMULA2 and still achieve the same level of test accuracy, while doing much better than the fully supervised baseline not regularized with state-of-the-art regularizers (20.26% error). Adversarial examples in some sense are the "worst case" scenario for models failing to perform well when evaluated with data off the manifold 3. Because Manifold Mixup only considers a subset of directions around data points (namely, those corresponding to interpolations), we would not expect the model to be robust to adversarial attacks which can consider any direction within an epsilon-ball of each example. At the same time, Manifold Mixup expands the set of points seen during training, so an intriguing hypothesis is that these overlap somewhat with the set of possible adversarial examples, which would force adversarial attacks to consider a wider set of directions, and potentially be more computationally expensive. To explore this we considered the Fast Gradient Sign Method (FGSM, BID6 which only requires a single gradient update and considers a relatively small subset of adversarial directions. The ing performance of Manifold Mixup against FGSM are given in TAB2 . A challenge in evaluating adversarial examples comes from the gradient masking problem in which a defense succeeds solely due to reducing the quality of the gradient signal. BID2 explored this issue in depth and proposed running an unbounded search for a large number of iterations to confirm the quality of the gradient signal. Our Manifold Mixup passed this sanity check (see Appendix F). While we found that Manifold Mixup greatly improved robustness to the FGSM attack, especially over Input Mixup , we found that Manifold Mixup did not significantly improve robustness against the stronger iterative projected gradient descent (PGD) attack . An important question is what kinds of feature combinations are being explored when we perform mixup in the hidden layers as opposed to linear interpolation in visible space. To provide a qualita- tive study of this, we trained a small decoder convnet (with upsampling layers) to predict an image from the Manifold Mixup classifier's hidden representation (using a simple squared error loss in the visible space). We then performed mixup on the hidden states between two random examples, and ran this interpolated hidden state through the convnet to get an estimate of what the point would look like in input space. Similarly to earlier on auto-encoders BID3, we found that these interpolated h points corresponded to images with a blend of the features from the two images, as opposed to the less-semantic pixel-wise blending ing from Input Mixup as shown in FIG4 and FIG5. Furthermore, this justifies the training objective for examples mixed-up in the hidden layers: most of the interpolated points correspond to combinations of semantically meaningful factors, thus leading to the more training samples; and none of the interpolated points between objects of two different categories A and B correspond to a third category C, thus justifying a training target which gives 0 probability on all the classes except A and B. Deep neural networks often give incorrect yet extremely confident predictions on data points which are unlike those seen during training. This problem is one of the most central challenges in deep learning both in theory and in practice. We have investigated this from the perspective of the representations learned by deep networks. In general, deep neural networks can learn representations such that real data points are widely distributed through the space and most of the area corresponds to high confidence classifications. This has major downsides in that it may be too easy for the network to provide high confidence classification on points which are off of the data manifold and also that it may not provide enough incentive for the network to learn highly discriminative representations. We have presented Manifold Mixup, a new algorithm which aims to improve the representations learned by deep networks by encouraging most of the hidden space to correspond to low confidence classifications while concentrating the hidden states for real examples onto a lower dimensional subspace. We applied Manifold Mixup to several tasks and demonstrated improved test accuracy and dramatically improved test likelihood on classification, better robustness to adversarial examples from FGSM attack, and improved semi-supervised learning. Manifold Mixup incurs virtually no additional computational cost, making it appealing for practitioners. We conducted experiments using a generated synthetic dataset where each image is deterministically rendered from a set of independent factors. The goal of this experiment is to study the impact of input mixup and an idealized version of Manifold Mixup where we know the true factors of variation in the data and we can do mixup in exactly the space of those factors. This is not meant to be a fair evaluation or representation of how Manifold Mixup actually performs -rather it's meant to illustrate how generating relevant and semantically meaningful augmented data points can be much better than generating points which are far off the data manifold. We considered three tasks. In Task A, we train on images with angles uniformly sampled between (-70 •, -50 •) (label 0) with 50% probability and uniformly between (50•, 80 •) (label 1) with 50% probability. At test time we sampled uniformly between (-30 •, -10 •) (label 0) with 50% probability and uniformly between (10 •, 30 •) (label 1) with 50% probability. Task B used the same setup as Task A for training, but the test instead used (-30 •, -20 •) as label 0 and (-10 •, 30 •) as label 1. In Task C we made the label whether the digit was a "1" or a "7", and our training images were uniformly sampled between (-70 •, -50 •) with 50% probability and uniformly between (50•, 80•) with 50% probability. The test data for Task C were uniformly sampled with angles from (-30•, 30 DISPLAYFORM0 The examples of the data are in FIG6 and are in table 4. In all cases we found that Input Mixup gave some improvements in likelihood but limited improvements in accuracy -suggesting that the even generating nonsensical points can help a classifier trained with Input Mixup to be better calibrated. Nonetheless the improvements were much smaller than those achieved with mixing in the ground truth attribute space. Figure 6 : An experiment on a network trained on the 2D spiral dataset with a 2D bottleneck hidden state in the middle of the network (the same setup as 1). Noise refers to gaussian noise in the bottleneck layer, dropout refers to dropout of 50% in all layers except the bottleneck itself (due to its low dimensionality), and batch normalization refers to batch normalization in all layers. This shows that the effect of concentrating the hidden states for each class and providing a broad region of low confidence between the regions is not accomplished by the other regularizers. We have found significant improvements from using Manifold Mixup, but a key question is whether the improvements come from changing the behavior of the layers before the mixup operation is applied or the layers after the mixup operation is applied. This is a place where Manifold Mixup and Input Mixup are clearly differentiated, as Input Mixup has no "layers before the mixup operation" to change. We conducted analytical experimented where the representations are low-dimensional enough to visualize. More concretely, we trained a fully connected network on MNIST with two fully-connected leaky relu layers of 1024 units, followed by a 2-dimensional bottleneck layer, followed by two more fully-connected leaky-relu layers with 1024 units. We then considered training with no mixup, training with mixup in the input space, and training only with mixup directly following the 2D bottleneck. We consistently found that Manifold Mixup has the effect of making the representations much tighter, with the real data occupying more specific points, and with a more well separated margin between the classes, as shown in FIG7 C SUPERVISED REGULARIZATION For supervised regularization we considered architectures within the PreActResNet family: PreActResNet18, PreActResNet34, and PreActResNet152. When using Manifold Mixup, we selected the layer to perform mixing uniformly at random from a set of eligible layers. In our experiments on PreActResNets in TAB2, for Manifold Mixup, our eligible layers for mixing were: the input layer, the output from the first resblock, and the output from the second resblock. For PreActResNet18, the first resblock has four layers and the second resblock has four layers. For PreActResNet34, the first resblock has six layers and the second resblock has eight layers. For PreActResNet152, the first resblock has 9 layers and the second resblock has 24 layers. Thus the mixing is often done fairly deep in the network, for example in PreActResNet152 the output of the second resblock is preceded by a total of 34 layers (including the initial convolution which is not in a resblock). For Manifold Mixup All layers in Table 1a, our eligible layers for mixing were: the input layer, the output from the first resblock, and the output from the second resblock, and the output from the third resblock. We trained all models for 1200 epochs and dropped the learning rates by a factor of 0.1 at 400 epochs and 800 epochs. Table 6 presents for SVHN dataset with PreActResNet18 architecture. In Figure 9 and FIG0, we present the training loss (Binary cross entropy) for Cifar10 and Cifar100 datasets respectively. We observe that performing Manifold Mixup in higher layers allows the train loss to go down faster as compared against the Input Mixup. This is consistent with the demonstration in FIG0: Input mixup can suffer from underfitting since the interpolation between two examples can intersect with a real example. In Manifold Mixup the hidden states in which the interpolation is performed, are learned, hence during the course of training they can evolve in such a way that the aforementioned intersection issue is avoided. We present the procedure for Semi-supervised Manifold Mixup and Semi-supervised Input Mixup in Algorithms 1 and 3 respectively. Algorithm 1 Semi-supervised Manifold Mixup. f θ: Neural Network; M anif oldM ixup: Manifold Mixup Algorithm 2; D L: set of labelled samples; D U L: set of unlabelled samples; π: consistency coefficient (weight of unlabeled loss, which is ramped up to increase from zero to its max value over the course of training); N: number of updates;ỹ i: Mixedup labels of labelled samples;ŷ i: predicted label of the labelled samples mixed at a hidden layer; y j: Psuedolabels for unlabelled samples;ỹ j: Mixedup Psuedolabels of unlabelled samples;ŷ j predicted label of the unlabelled samples mixed at a hidden layer DISPLAYFORM0 Cross Entropy loss 6: DISPLAYFORM1 g ← ∇ θ L (Gradients of the minibatch Loss)12:θ ← Update parameters using gradients g (e.g. SGD) 13: end while DISPLAYFORM2 Sample labeled batch 5: DISPLAYFORM3 Compute Pseudolabels 9: DISPLAYFORM4 10: DISPLAYFORM5 g ← ∇ θ L Gradients of the minibatch Loss 13:θ ← Update parameters using gradients g (e.g. SGD) 14: end while Table 5: Models trained on the normal CIFAR-100 and evaluated on a test set with novel deformations. Manifold Mixup (ours) consistently allows the model to be more robust to random shearing, rescaling, and rotation even though these deformations were not observed during training. For the rotation experiment, each image is rotated with an angle uniformly sampled from the given range. Likewise the shearing is performed with uniformly sampled angles. Zooming-in refers to take a bounding box at the center of the image with k% of the length and k% of the width of the original image, and then expanding this image to fit the original size. Likewise zooming-out refers to drawing a bounding box with k% of the height and k% of the width, and then taking this larger area and scaling it down to the original size of the image (the padding outside of the image is black). 2.37 Input Mixup (α = 1.5) 2.41 Manifold Mixup (α = 1.5) 1.92 Manifold Mixup (α = 2.0) 1.90 Figure 9: CIFAR-10 train set Binary Cross Entropy Loss (BCE) on Y-axis using PreActResNet18, with respect to training epochs (X-axis). The numbers in {} refer to the resblock after which Manifold Mixup is performed. The ordering of the losses is consistent over the course of training: Manifold Mixup with gradient blocked before the mixing layer has the highest training loss, followed by Input Mixup. The lowest training loss is achieved by mixing in the deepest layer, which is highly consistent with Section 3 which suggests that having more hidden units can help to prevent underfitting. FIG0: CIFAR-100 train set Binary Cross Entropy Loss (BCE) on Y-axis using PreActResNet50, with respect to training epochs (X-axis). The numbers in {} refer to the resblock after which Manifold Mixup is performed. The lowest training loss is achieved by mixing in the deepest layer. We use the WideResNet28-2 architecture used in and closely follow their experimental setup for fair comparison with other Semi-supervised learning algorithms. We used SGD with momentum optimizer in our experiments. For Cifar10, we run the experiments for 1000 epochs with initial learning rate is 0.1 and it is annealed by a factor of 0.1 at epoch 500, 750 and 875. For SVHN, we run the experiments for 200 epochs with initial learning rate is 0.1 and it is annealed by a factor of 0.1 at epoch 100, 150 and 175. The momentum parameter was set to 0.9. We used L2 regularization coefficient 0.0005 and L1 regularization coefficient 0.001 in our experiments. We use the batch-size of 100.The data pre-processing and augmentation in exactly the same as in . For CIFAR-10, we use the standard train/validation split of 45,000 and 5000 images for training and validation respectively. We use 4000 images out of 45,000 train images as labelled images for semi-supervised learning. For SVHN, we use the standard train/validation split with 65932 and 7325 images for training and validation respectively. We use 1000 images out of 65932 images as labelled images for semi-supervised learning. We report the test accuracy of the model selected based on best validation accuracy. For supervised loss, we used α (of λ ∼ Beta(α, α)) from the set {0.1, 0.2, 0.3... 1.0} and found 0.1 to be the best. For unsupervised loss, we used α from the set {0.1, 0.5, 1.0, 1.5, 2.0. 3.0, 4.0} and found 2.0 to be the best. The consistency coefficient is ramped up from its initial value 0.0 to its maximum value at 0.4 factor of total number of iterations using the same sigmoid schedule of . For CIFAR-10, we found max consistency coefficient = 1.0 to be the best. For SVHN, we found max consistency coefficient = 2.0 to be the best. When using Manifold Mixup, we selected the layer to perform mixing uniformly at random from a set of eligible layers. In our experiments on WideResNet28-2 in TAB1, our eligible layers for mixing were: the input layer, the output from the first resblock, and the output from the second resblock. We ran the unbounded projected gradient descent (PGD) sanity check suggested in BID2. We took our trained models for the input mixup baseline and manifold mixup and we ran PGD for 200 iterations with a step size of 0.01 which reduced the mixup model's accuracy to 1% and reduced the Manifold Mixup model's accuracy to 0%. This is direct evidence that our defense did not improve primarily as a of gradient masking. The Fast Gradient Sign Method (FGSM) BID6 is a simple one-step attack that produces x = x + ε sgn(∇xL(θ, x, y)). The recent literature has suggested that regularizing the discriminator is beneficial for training GANs (; BID7 b). In a similar vein, one could add mixup to the original GAN training objective such that the extra data augmentation acts as a beneficial regularization to the discriminator, which is what was proposed in. Mixup proposes the following objective 4: max DISPLAYFORM0 where x1, x2 can be either real or fake samples, and λ is sampled from a U nif orm(0, α). Note that we have used a function y(λ; x1, x2) to denote the label since there are four possibilities depending on x1 and x2: DISPLAYFORM1 if x1 is real and x2 is fake 1 − λ, if x1 is fake and x2 is real 0, if both are fake 1, if both are realIn practice however, we find that it did not make sense to create mixes between real and real where the label is set to 1, (as shown in equation 9), since the mixup of two real examples in input space is not a real example. So we only create mixes that are either real-fake, fake-real, or fake-fake. Secondly, instead of using just the equation in 8, we optimize it in addition to the regular minimax GAN equations: DISPLAYFORM2 Using similar notation to earlier in the paper, we present the manifold mixup version of our GAN objective in which we mix in the hidden space of the discriminator: DISPLAYFORM3 where h k (·) is a function denoting the intermediate output of the discriminator at layer k, and d k (·) the output of the discriminator given input from layer k. The layer k we choose the sample can be arbitrary combinations of the input layer (i.e., input mixup), or the first or second resblocks of the discriminator, all with equal probability of selection. We run some experiments evaluating the quality of generated images on CIFAR10, using as a baseline JSGAN with spectral normalization (b) (our configuration is almost identical to theirs). Results are averaged over at least three runs 5. From these , the best-performing mixup experiments (both input and Manifold Mixup) is with α = 0.5, with mixing in all layers (both resblocks and input) achieving an average Inception / FID of 8.04 ± 0.08 / 21.2 ± 0.47, input mixup achieving 8.03 ± 0.08 / 21.4 ± 0.56, for the baseline experiment 7.97 ± 0.07 / 21.9 ± 0.62. This suggests that mixup acts as a useful regularization on the discriminator, which is even further improved by Manifold Mixup. See FIG0 for the full set of experimental An essential motivation behind manifold mixup is that because the network learns the hidden states, it can do so in such a way that the interpolations between points are consistent. Section 3 characterized this for hidden states with any number of dimensions and FIG0 showed how this can occur on the 2d spiral dataset. Our goal here is to discuss concrete examples to illustrate what it means for the interpolations to be consistent. If we consider any two points, the interpolated point between them is based on a sampled λ and the soft-target for that interpolated point is the targets interpolated with the same λ. So if we consider two points A,B which have the same label, it is apparent that every point on the line between A and B should have that same label with 100% confidence. If we consider two points A,B with different labels, then the point which is halfway between them will be given the soft-label of 50% the label of A and 50% the label of B (and so on for other λ values).It is clear that for many arrangements of data points, it is possible for a point in the space to be reached through distinct interpolations between different pairs of examples, and reached with different λ values. Because the learned model tries to capture the distribution p(y|h), it can only assign a single distribution over the label values to a single particular point (for example it could say that a point is 100% label A, or it could say that a point is 50% label A and 50% label B). Intuitively, these inconsistent soft-labels at interpolated points can be avoided if the states for each class are more concentrated and classes vary along distinct dimensions in the hidden space. The theory in Section 3 characterizes exactly what this concentration needs to be: that the representations for each class need to lie on a subspace of dimension equal to "number of hidden dimensions" -"number of classes" + 1.Figure 12: We consider a binary classification task with four data points represented in a 2D hidden space. If we perform mixup in that hidden space, we can see that if the points are laid out in a certain way, two different interpolations can give inconsistent soft-labels (left and middle). This leads to underfitting and high loss. When training with manifold mixup, this can be explicitly avoided because the states are learned, so the model can learn to produce states for which all interpolations give consistent labels, an example of which is seen on the right side of the figure. When we refer to flattening, we mean that the class-specific representations have reduced variability in some directions. Our analysis in this section makes this more concrete. We trained an MNIST classifier with a hidden state bottleneck in the middle with 12 units (intentionally selected to be just slightly greater than the number of classes). We then took the representation for each class and computed a singular value decomposition (FIG0 and FIG0 and we also computed an SVD over all of the representations together ( FIG0). Our architecture contained three hidden layers with 1024 units and LeakyReLU activation, followed by a bottleneck representation layer (with either 12 or 30 hidden units), followed by an additional four hidden layers each with 1024 units and LeakyReLU activation. When we performed Manifold Mixup for our analysis, we only performed mixing in the bottleneck layer, and used a beta distribution with an alpha of 2.0. Additionally we performed another experiment FIG0 where we placed the bottleneck representation layer with 30 units immediately following the first hidden layer with 1024 units and LeakyReLU activation. We found that Manifold Mixup had a striking effect on the singular values, with most of the singular values becoming much smaller. Effectively, this means that the representations for each class have variance in fewer directions. While our theory in Section 3 showed that this flattening must force each classes representations onto a lower-dimensional subspace (and hence an upper bound on the number of singular values) but this explores how this occurs empirically and does not require the number of hidden dimensions to be so small that it can be manually visualized. In our experiments we tried using 12 hidden units in the bottleneck FIG0 as well as 30 hidden units FIG0 in the bottleneck. Our from this experiment are unequivocal: Manifold Mixup dramatically reduces the size of the smaller singular values for each classes representations. This indicates a flattening of the class-specific representations. At the same time, the singular values over all the representations are not changed in a clear way FIG0 ), which suggests that this flattening occurs in directions which are distinct from the directions occupied by representations from other classes, which is the same intuition behind our theory. Moreover, FIG0 shows that when the mixing is performed earlier in the network, there is still a flattening effect, though it is weaker than in the later layers, and again Input Mixup has an inconsistent effect. Figure 13: SVD on the class-specific representations in a bottleneck layer with 12 units following 3 hidden layers. For the first singular value, the value (averaged across the plots) is 50.08 for the baseline, 37.17 for Input Mixup, and 43.44 for Manifold Mixup (these are the values at x=0 which are cutoff). We can see that the class-specific SVD leads to singular values which are dramatically more concentrated when using Manifold Mixup with Input Mixup not having a consistent effect. We compare the performance of Manifold Mixup using different values of hyper-parameter α by training a PreActResNet18 network on Cifar10 dataset, as shown in TAB6. Manifold Mixup outperformed Input Mixup for all alphas in the set (0.5, 1.0, 1.2, 1.5, 1.8, 2.0) -indeed the lowest for Manifold Mixup is better than the worst with Input Mixup. Note that Input Mixup's deteriorate when using an alpha that is too large, which is not seen with manifold mixup. In this section, we discuss which layers are a good candidate for mixing in the Manifold Mixup algorithm. We evaluated PreActResNet18 models on CIFAR-10 and considered mixing in a subset of the layers, we ran for fewer epochs than in the Section 5.1 (making the accuracies slightly lower across the board), and we decided to fix the alpha to 2.0 as we did in the the Section 5.1. We considered different subsets of layers to mix in, with 0 referring to the input layer, 1/2/3 referring to the output of the 1st/2nd/3rd resblocks respectively. For example 0,2 refers to mixing in the input layer and the output of the 2nd resblock. {} refers to no mixing. The are presented in TAB7 Essentially, it helps to mix in more layers, except for the later layers which hurts the test accuracy to some extent -which is consistent with our theory in Section 3: the theory in Section 3 assumes that the part of the network after mixing is a universal approximator, hence, there is a sensible case to be made for not mixing in the very last layers.

Represent the TLDR for finding the corresponding scientific article

A method for learning better representations, that acts as a regularizer and despite its no significant additional computation cost , achieves improvements over strong baselines on Supervised and Semi-supervised Learning tasks.

represent the tldr for finding the corresponding scientific article

Examples:

"We proposed a universal method which can be used in the data preprocessing stage to generate the more meaningful topic that better represents the given document" == "The topic modeling discovers the latent topic probability of given the text documents. To generate the more meaningful topic that better represents the given document, we proposed a universal method which can be used in the data preprocessing stage. The method consists of three steps. First, it generates the word/word-pair from every single document. Second, it applies a two way parallel TF-IDF algorithm to word/word-pair for semantic filtering. Third, it uses the k-means algorithm to merge the word pairs that have the similar semantic meaning. Experiments are carried out on the Open Movie Database (OMDb), Reuters Dataset and 20NewsGroup Dataset and use the mean Average Precision score as the evaluation metric. Comparing our with other state-of-the-art topic models, such as Latent Dirichlet allocation and traditional Restricted Boltzmann Machines. Our proposed data preprocessing can improve the generated topic accuracy by up to 12.99\%. How the number of clusters and the number of word pairs should be adjusted for different type of text document is also discussed. After millennium, most collective information are digitized to form an immense database distributed across the Internet. Among all, text-based knowledge is dominant because of its vast availability and numerous forms of existence. For example, news, articles, or even Twitter posts are various kinds of text documents. For the human, it is difficult to locate one's searching target in the sea of countless texts without a well-defined computational model to organize the information. On the other hand, in this big data era, the e-commerce industry takes huge advantages of machine learning techniques to discover customers' preference. For example, notifying a customer of the release of "Star Wars: The Last Jedi" if he/she has ever purchased the tickets for "Star Trek Beyond"; recommending a reader "A Brief History of Time" from Stephen Hawking in case there is a "Relativity: The Special and General Theory" from Albert Einstein in the shopping cart on Amazon. The content based recommendation is achieved by analyzing the theme of the items extracted from its text description. Topic modeling is a collection of algorithms that aim to discover and annotate large archives of documents with thematic information BID0. Usually, general topic modeling algorithms do not require any prior annotations or labeling of the document while the abstraction is the output of the algorithms. Topic modeling enables us to convert a collection of large documents into a set of topic vectors. Each entry in this concise representation is a probability of the latent topic distribution. By comparing the topic distributions, we can easily calculate the similarity between two different documents. BID25 Some topic modeling algorithms are highly frequently used in text-mining BID13, preference recommendation BID27 and computer vision BID28. BID0 Many of the traditional topic models focus on latent semantic analysis with unsupervised learning. Latent Semantic Indexing (LSI) BID11 applies Singular-Value Decomposition (SVD) BID6 to transform the term-document matrix to a lower dimension where semantically similar terms are merged. It can be used to report the semantic distance between two documents, however, it does not explicitly provide the topic information. The Probabilistic Latent Semantic Analysis (PLSA) BID9 model uses maximum likelihood estimation to extract latent topics and topic word distribution, while the Latent Dirichlet Allocation (LDA) BID1 model performs iterative sampling and characterization to search for the same information. The availability of many manually categorized online documents, such as Internet Movie Database (IMDb) movie review Inc., Wikipedia articles, makes the training and testing of topics models possible. All of the existing workds are based on the bag-of-words model, where a document is considered as a collection of words. The semantic information of words and interaction among objects are assumed to be unknown during the model construction. Such simple representation can be improved by recent research advances in natural language processing and word embedding. In this paper, we will explore the existing knowledge and build a topic model using explicit semantic analysis. The work studies the best data processing and feature extraction algorithms for topic modeling and information retrieval. We investigate how the available semantic knowledge, which can be obtained from language analysis or from existing dictionary such as WordNet, can assist in the topic modeling. Our main contributions are:• We redesign a new topic model which combines two types of text features to be the model input.• We apply the numerical statistic algorithm to determine the key elements for each document dynamically.• We apply a vector quantization method to merge and filter text unit based on the semantic meaning.• We significantly improve the accuracy of the prediction using our proposed model. The rest of the paper is structured as follows: In Section 2, we review the existing methods, from which we got the inspirations. This is followed in Section 3 by details about our topic models. Section 4 describes our experimental steps and evaluate the . Finally, Section 5 concludes this work. Many topic models have been proposed in the past decades. This includes LDA, Latent Semantic Analysis(LSA), word2vec, and Restricted Boltzmann Machine (RBM), etc. In this section, we will compare the pros and cons of these topic models. LDA was one of the most widely used topic models. LDA introduces sparse Dirichlet prior distributions over document-topic and topic-word distributions, encoding the intuition that documents cover a small number of topics and that topics often use a small number of words. BID1 LSA was another topic modeling technique which is frequently used in information retrieval. LSA learned latent topics by performing a matrix decomposition (SVD) on the term-document matrix. BID4 In practice, training the LSA model is faster than training the LDA model, but the LDA model is more accurate than the LSA model. Traditional topic models did not consider the semantic meaning of each word and cannot represent the relationship between different words. Word2vec can be used for learning high-quality word vectors from huge data sets with billions of words, and with millions of words in the vocabulary. BID14 During the training, the model generated word-context pairs by applying a sliding window to scan through a text corpus. Then the word2vec model trained word embeddings using word-context pairs by using the continuous bag of words (CBOW) model and the skip-gram model. BID15 The generated word vectors can be summed together to form a semantically meaningful combination of both words. Moreover, there were a lot of extensions of the word2vec model, such as paragraph2vec and LDA2vec. Paragraph2vec was an unsupervised framework that learned continuous distributed vector representations for pieces of texts. The training of the paragraph2vec model is based on the similar idea as the word2vec. One of the advantages of the generated paragraph vector was that they take into consideration the word order. BID12 LDA2vec focused on utilizing document-wide feature vectors while simultaneously learning continuous document weights loading onto topic vectors. LDA2vec embedded both words and document vectors into the same space and train both representations simultaneously. BID18 RBM was proposed to extract low-dimensional latent semantic representations from a large collection of documents BID8. The architecture of the RBMs is an undirected bipartite graphic, in which word-count vectors were modeled as Softmax input units and the output units were usually binary units. The RBM model can be generalized much better than LDA in terms of both log-probability on the document and the retrieval accuracy. A deeper structure of neural network was developed based on stacked RBMs, which was the Deep Belief Network (DBN). In BID7, the input layer was the same as RBM mentioned above, other layers are all binary units. 3.1 WORD PAIR BASED RBM MODEL Current RBM model for topic modeling uses Bag of Words approach. Each visible neuron represents the number of appearance of a dictionary word. We believe that the order of the words also exhibits rich information, which is captured by the bag of words approach. Our hypothesis is that including word pairs (with specific dependencies) helps to improve topic modeling. In this work, Stanford natural language parser BID3 BID19 is used to analyze sentences in both training and testing texts, and extract word pairs. For example, if we have the following sentence, by applying the word pair extraction, we will get 38 word pairs and one root pair as the FIG0.The strongest rain ever recorded in India shut down the financial hub of Mumbai, snapped communication lines, closed airports and forced thousands of people to sleep in their offices or walk home during the night, officials said today. The first part in each word pair segment represents the relationship between two words, such as det, nsubj, advmod, etc. And the number after each word gives its position of in the original sentence. The two words extracted in this way are not necessarily adjacent to each other, however, they are semantically related. Because each single word may have different combinations with other words, the total number of the word pairs will be much larger than the number of word in the training dataset. If we use all word pairs which are extracted from the training dataset, it will significantly increase the size of our dictionary and reduce the performance. So, we only keep the first 10000 most frequent word pairs to be our word pair dictionary. TF-IDF stands for the term frequency -inverse document frequency, and the TF-IDF weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. BID23 BID22 BID21 BID29 The equation of the TF-IDF is as following: DISPLAYFORM0 N umber of times term t appears in a docuemnt T otal number of terms in the document DISPLAYFORM1 DISPLAYFORM0 Term Frequency (TF), Equation 1, measures how frequently a term occurs in a document. Inverse document frequency (IDF), Equation 2, measures how important a term is. There are many words in the training dataset. If we use all words to build the training dictionary, it will contain a lot of high frequency but useless words, like "first" and "name". Beside removing stop word and processing word tense in the dataset, we also used the TF-IDF algorithm to filter the dataset. In addition, based on the original TF-IDF algorithm, we proposed a two-step TF-IDF processing method. This method is applied to keep the number of word pairs to a manageable size as shown in First, word pairs are generated and word-level TF-IDF is performed. The of word level TF-IDF is used as a filter and a word pair is kept only if the TF-IDF scores of both words are higher than the threshold (0.01). After that, we treat each word pair as a single unit, and the TF-IDF algorithm is applied to the word pairs and further filter out word pairs that are either too common or too rare. Even with the TF-IDF processing, the size of the word pair dictionary is still prohibitively large and selecting only the most frequent ones is a brute force approach. We further cluster semantically close word pairs to reduce the dictionary size. The semantic distance between two words is measured as the distance of their embedding vectors calculated using Googles word2vec model. The only issue is how to determine the cluster centrum. Our first approach is to use upper level words in WordNet BID17 BID16 as cluster centrum. Our hypothesis is that, since those upper-level words have broader meaning, a relatively small number of these words can provide good coverage of the semantical space. We first build a word level tree based on the relations specified in the WordNet. We then pick those words that are closest to the root as the cluster centrum. Words usually have multiple paths to the root. We first add those synsets that have only one path to the root into the graph, then iteratively examine the concepts that we have left and add the paths that grow the tree by as little as possible. Based on which path is selected, this approach can be further divided into min-path clustering and max-path clustering. The min-path clustering will add the shortest path and the max-path clustering will add the longest path to the graph The second approach is K-means clustering. By applying the K-mean algorithm, we group the embedding vector of words into K clusters. Then we use the index of each cluster to represent a group of words. Because the word embedding is a very dense space, the k-mean clustering does not give good Silhouette score BID20. Our original hypothesis is that the K-mean clustering based approach will not be as effective as the WordNet-based approach. However, our experimental show that this is not right. So, we pick the K-means clustering algorithm to be our final feature dictionary organization method. In our experiment, we generate the topic distribution for each document by using RBM model. Then we retrieve the top N documents by calculating Euclidean distance. Our proposed method is evaluated on 3 datasets: OMDb, Reuters, and 20NewsGroup. For the Reuters and 20NewsGroup dataset, we download them from BID2. The OMDb dataset is collected manually by using OMDb APIs (Fritz). All the datasets are divided into three sub-dataset: training, validation, and testing. The split ratio is 70:10:20. For each dataset, a 5-fold cross-validation is applied.• OMDb stands for The Open Movie Database. The training dataset contains 6043 movie descriptions, the validation dataset contains 863 movie descriptions and the testing dataset contains 1727 movie descriptions. We define the class for each movie by applying K-means clustering on category information. The value of K is set to 20.• Reuters, these documents appeared on the Reuters newswire in 1987 and were manually classified by personnel from Reuters Ltd. There are 7674 documents in total which belongs to 8 classes. The training dataset contains 5485 news, the validation dataset contains 768 news and the testing dataset contains 1535 news.• 20NewsGroup, this dataset is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. The training dataset contains 13174 news, the validation dataset contains 1882 news and the testing dataset contains 3765 news. We use mean Average P recision (mAP) score to evaluate our proposed method. It is a score to evaluate the information retrieval quality. This evaluation method considers the effect of order in the information retrieval . If the relational is shown in the front position, the score will tend to 1; if the relational is shown in the back position, the score will tend to 0. mAP 1, mAP 3, mAP 5, and mAP 10 are used to evaluate the retrieval performance. For each document, we retrieve 1, 3, 5, and 10 documents whose topic vectors have the smallest Euclidean distance with that of the query document. The documents are considered as relevant if they share the same class label. Before we calculate the mAP, we need to calculate the Average P recision (AveP) for each document first. The equation of AveP is described below. DISPLAYFORM0 where rel(k) is an indicator function equaling 1 if the item at rank k is a relevant document, 0 otherwise. BID26 Note that the average is over all relevant documents and the relevant documents not retrieved get a precision score of zero. The equation of the mean Average P recision (mAP) score is as following: DISPLAYFORM1 where Q indicates the total number of queries. In this experiment, the total feature size is a fixed number. Then, we compare the performance between two RBM models. One of them only considers words as the input feature, while the other has combined words and word pairs as the input feature. The total feature size varies from 10500, 11000, 11500, 12000, 12500, 15000. For the word/word pair combined RBM model, the number of word is 10000, and the number of word pairs is varied to meet the total feature requirement. Both models are applied to the OMDb dataset, and the are shown in FIG2 and TAB0, the word/word pair combined model almost always performs better than the word-only model. For the mAP 1, the mAP 5 and the mAP 10, the most significant improvement shown in Feature Number = 11000, about 10.48%, 7.97% and 9.83%. For the mAP 3, the most significant improvement shown in Feature Number = 12000, about 9.35%. The two models are further applied on the Reuters dataset, and the are shown in FIG3 and TAB1. Again the word/word pair combined model outperforms the word-only model almost all the time. For the mAP 1, the most significant improvement happens when Feature Number = 12500. Under this feature size, the combined model improves the mAP score by approximately 1.05%. For the mAP 3, the mAP 5 and the mAP 10, the most significant improvement happens when Feature Number = 15000. Under this feature size, the combined model gives about 1.11%, 1.02% and 0.89% improvement. The for 20NewsGroup dataset are shown in Figure 5 and Table 3. Similar to previous two datasets, all the from word/word pair combined model are better than the word-only model. For the mAP 1 the mAP 3, the mAP 5 and the mAP 10, the most significant improvement happens when Feature Number = 11500. And the improvements are about 10.40%, 11.91%, 12.46% and 12.99%.Observing the for all three datasets, we found that they all reflect the same pattern. The word/word pair combined model consistently outperforms the word only model, given that both of them are constructed based on the same number of input features. In the second experiment, we focus on how the different K values affect the effectiveness of the generated word pairs in therms of their ability of topic modeling. First, we average the performance word/word pair combination all K value. The potential K values are 100, 300, 500, 800 and 1000. Then we compare the mAP between our model and the baseline model, which consists of word only input features. The OMDb dataset are shown in Table 4. As we can observe, all the K values give us better performance than the baseline. The most significant improvement shown in K = 100, which are about 2.41%, 2.15%, 1.46% and 4.46% for mAP 1, 3, 5, and 10 respectively. The of Reuters dataset are shown in FIG5 and Table 5. When the K value is greater than 500, all mAP for word/word pair combination model are better than the baseline. Because the mAP score for Reuters dataset in original model is already very high almost all of them higher than 0.9, it is hard to get the improvement as large as OMDb dataset. For the mAP 1, the most significant improvement happens when K = 500, which is 0.31%. For the mAP 5, the mAP 5 and the mAP 10, the most significant improvement shown in K = 800, about 0.50%, 0.38% and 0.42%.The for 20NewsGroup dataset are shown in FIG6 and Table 6. Similar to the Reuters dataset, when the K value is greater than 800, all mAP score for word/word pair combination model are better than the baseline. For the mAP 1, 3, 5, and 10, the most significant im- provements about 2.82%, 2.90%, 3.2% and 3.33% respectively, and they all happened when K = 1000.In summary, a larger K value can give us a better . As we can see from the Reuters dataset and the 20NewsGroup dataset, when K is greater than 800, the combined model outperforms the baseline model in all four mAP evaluation scores. However, the best appear when K = 800 or K = 1000. For the OMDb dataset, because the mAP score for the baseline model is not very high, the combined model gives better than the baseline model with any K value that we tested. In this experiment, we compare different word pair generation algorithms with the baseline. Similar to previous experiments, the baseline is the word-only RBM model whose input consists of the 10000 most frequent words. The "semantic" word pair generation is the method we proposed in this paper. By applying the idea from the skip-gram BID15 algorithm, we generate the word pairs from each word's adjacent neighbor, and we call it "N-gram" word pair generation. And the window size we used in here is N = 2. For the Non-K word pair generation, we use the same algorithm as the semantic except that no K-means clustering is applied on the generated word pairs. The first thing we observe from From the TAB2 is that both "semantic" word pair generation and "Non-K" word pair generation give us better mAP score than the baseline; however, the mAP score of the semantic generation is slightly higher than the mAP score of the Non-K generation. This is because, although both Non-K and semantic techniques extract word pairs using natural language processing, without the K-means clustering, semantically similar pairs will be considered separately. Hence there will be lots of redundancies in the input space. This will either increase the size of the input space, or, in order to control the input size, reduce the amount of information captured by the input set. The K-mean clustering performs the function of compress and feature extraction. The second thing that we observe is that, for the N-gram word pair gener-ation, its mAP score is even lower than the baseline. Beside the OMDb dataset, other two datasets show the same pattern. This is because the semantic model extracts word pairs from natural language processing, therefore those word pairs have the semantic meanings and grammatical dependencies. However, the N-gram word pair generation simply extracts words that are adjacent to each other. When introducing some meaningful word pairs, it also introduces more meaningless word pairs at the same time. These meaningless word pairs act as noises in the input. Hence, including word pairs without semantic importance does not help to improve the model accuracy. In this paper, we proposed a few techniques to processes the dataset and optimized the original RBM model. During the dataset processing part, first, we used a semantic dependency parser to extract the word pairs from each sentence of the text document. Then, by applying a two way parallel TF-IDF processing, we filtered the data in word level and word pair level. Finally, Kmeans clustering algorithm helped us merge the similar word pairs and remove the noise from the feature dictionary. We replaced the original word only RBM model by introducing word pairs. At the end, we showed that proper selection of K value and word pair generation techniques can significantly improve the topic prediction accuracy and the document retrieval performance. With our improvement, experimental have verified that, compared to original word only RBM model, our proposed word/word pair combined model can improve the mAP score up to 10.48% in OMDb dataset, up to 1.11% in Reuters dataset and up to 12.99% in the 20NewsGroup dataset."

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: represent the scientific article to find a short tldr.

Question: Sometimes SRS (Stereotactic Radio Surgery) requires using sphere packing on a Region of Interest (ROI) such as cancer to determine a treatment plan. We have developed a sphere packing algorithm which packs non-intersecting spheres inside the ROI. The region of interest in our case are those voxels which are identified as cancer tissues. In this paper, we analyze the rotational invariant properties of our sphere-packing algorithm which is based on distance transformations. Epsilon-Rotation invariant means the ability to arbitrary rotate the 3D ROI while keeping the volume properties remaining (almost) same within some limit of epsilon. The applied rotations produce spherical packing which remains highly correlated as we analyze the geometrically properties of sphere packing before and after the rotation of the volume data for the ROI. Our novel sphere packing algorithm has high degree of rotation invariance within the range of +/- epsilon. Our method used a shape descriptor derived from the values of the disjoint set of spheres form the distance-based sphere packing algorithm to extract the invariant descriptor from the ROI. We demonstrated by implementing these ideas using Slicer3D platform available for our research. The data is based on sing MRI Stereotactic images. We presented several performance on different benchmarks data of over 30 patients in Slicer3D platform. Sometimes SRS (Stereotactic Radio Surgery) requires using sphere packing on a Region of Interest (ROI) such as cancer to determine a treatment plan. We have developed a sphere packing algorithm which packs non-intersecting spheres inside the ROI. The region of interest in our case are those voxels which are identified as cancer tissues. In this paper, we analyze the rotational invariant properties of our spherepacking algorithm which is based on distance transformations. e-Rotation invariant means the ability to arbitrary rotate the 3D ROI while keeping the volume properties remaining (almost) same within some limit of e. The applied rotations produce spherical packing which remains highly correlated as we analyze the geometrically properties of sphere packing before and after the rotation of the volume data for the ROI. Our novel sphere packing algorithm has high degree of rotation invariance within the range of ± e. Our method used a shape descriptor derived from the values of the disjoint set of spheres form the distance-based sphere packing algorithm to extract the invariant descriptor from the ROI. We demonstrated by implementing these ideas using Slicer3D platform available for our research. The data is based on sing MRI Stereotactic images. We presented several performance on different benchmarks data of over 30 patients in Slicer3D platform. Rotation Invariant, Slicer3D, Sphere Packing, Distance Transformation, Stereotactic. In several applications such as inspection of tumor or interacting with portion of a 3D volume data, the ROI could be rotated at arbitrary angles. If a sphere packing algorithm is used before and after such rotation, then rotational invariance suggests that there might be high correlation between spheres found by our sphere packing algorithm before and after the rotation. Defining correspondences between the original and rotated ROIs is an important task that could be solved by spheres' descriptors. If these descriptors are highly correlated, then we can anticipate that the ROIs might be similar as well. Li et al. stated that translation and scaling are easy compared to rotation. Rotation of a 3D volume data or 3D image involves simultaneous manipulation of three coordinates to maintain invariance. In the case of sphere packing, as we capture the ROI with nonintersecting spheres, the rotation invariance means that set of spheres will remain identical in size although their placement is expected to change under an arbitrary rotation. There are three major techniques to prove the rotation invariance: landmarking, rotation invariant features/shape extraction descriptor, and brute force rotation alignment. The landmarking is normally carried out by following two methods, domain specific landmarking and generic landmarking . The domain specific landmarking accepts some fixed point in the image and does rotation with respect to that about an arbitrary axis. The generic landmarking method on the other hand, finds the major axes of the 3D/2D image and that can rotate the volume or image as a whole in carrying out the rotation. Because the size of the volume data can be typically large based on the size of the data, both these approaches require that large memory storage is available as the complete voxel information is required, and usually is time consuming. The brute force alignment method divides/degrades the object into large number of smaller parts and works with them for rotation. This method is time consuming, complex and complicated because parts have to be organized. The developed code for a particular shape in this method may only apply to the data in hand and may not be generalizable. Finally, Invariant feature/shape descriptor involves identification of certain invariant features (measurable quantities) that remains unaltered under rotations of the 3D image or volume data. The invariant features are indexed with a feature vector also known as shape signatures. Then, the optimal rotation can be defined by measuring model's similarities in terms of the distance such that the rotation invariant property would mean that these distance measures are as close to each other with certain limit before and after the rotation. There are literally many of rotation invariant features that been used in the past, including ratio of perimeter to area, fractal measures, circularity, min/max/mean curvature, and shape histograms, etc. Lin et al. and Yankov et al. use time series representation as a feature vector to match the 3D shapes to prove the rotation invariance. Based on our research, most of the studies have been used spherical harmonic method to map the features of objects into a unit sphere to prove the invariance under rotation (; ;). The spherical harmonic method does not always give accurate to distinguish between models since the internal parts of the 3D shapes may not fit in same sphere. Other researchers combined the spherical harmonic with spatial geometric moments . The most common graph method used is skeletons. The skeletons are based on medial axis. The medial axis of the 3D objects has been used as a shape descriptor in a number of researches;;;; ). However, this method is sensitive to noise and has a heavy computationally cost. In this paper, we considered the set of spheres as shape-descriptors and analyzed the sphere packing before and after the rotations and looked for the similarity measure. We aimed to show that set of spheres are invariant such that even if we rotate the image, the size of the spheres and center's distances are highly correlated. We used our sphere packing algorithm to pack non-intersecting spheres into the ROIs before and after rotations. As mentioned earlier, those spheres could provide invariant shape descriptor. After rotation the voxels will be populated with the new voxel orientation. Our shape descriptor provides a novel featureless method that doesn't depend on any specific feature or texture, instead is related to sphere packing generated by our sphere packing algorithm. Our method characterizes the 3D object similarity by the shape geometries of the sphere packing, and sphere's correspondence with one another and their spatial relationships. In this paper, we show that our previous work for sphere packing can be used to show the invariance under rotation since our algorithm can describe volumetric shapes more succinctly than voxel representation. In this work, the spheres packing together with the radiuses and centers functions provided an shape descriptor, a novel approach for characterization and compression of shape information for 3D volume and voxel data. In Stereotactic radio surgery, tumors are irradiated by beams of high-energy waves. It is a challenge during cancer treatment planning to provide minimal damage to healthy tissue around the tumors that get exposed to the radiation and still radiate cancerous cells. Our goal using sphere packing is to arrange beams on "spheres" in a way that hit the unhealthy tissue and leave the healthy tissue intact. A key geometric problem in Stereotactic radio surgery planning is to fill a 3D irregular-tumor shape (ROI) with disjointed spheres. We use the spheres packing to represent the 3D object, so this method of representation is called region-based descriptor since it is based on regions. In one of our work, Sphere Packing algorithm is used based on the maximum Euclidean distance has been studied and implemented in Slicer3D using medical imaging . Sphere packing problem is heuristically solved by using Euclidean maximum distance. The solution is to find a set of nonintersecting spheres that used greedy method and can be called largest sphere first. Each sphere is characterized by its radius and center. The size of the regions can be controlled depending on the treatment planning required size. Also, in our implementation, the density of the volume coverage can be customized such as we did in , we used 50%-90% of the density. This means 50% to 90% (or theoretically any amount up to 100%) of the ROI is covered disjoint spheres which our algorithm finds. Of course, more the coverage, more time is taken by the algorithm to find all the spheres satisfying the user selected criteria. Generally for all patients, 50% coverage takes up to 25 minutes, and 90% takes minimum of 7 hours and maximum of 72 hours. Our algorithm for the sphere packing is defined as a set of n unequal spheres and object P of a bounding box B. Each sphere = {1,2, … . .,} is characterized by its radius. and center.. The goal of this algorithm is to pack sets of disjoint spheres inside the ROI providing certain coverage. Our strategy is as follows: A uniform grid (voxelization) is used to calculate the maximum distance of each voxel to the 3D object boundary. Then, use the maximum distance to be the radius of the first sphere and the location to be the sphere center. Iteratively, we extract new spheres each time and recalculate the distances based on the following constraints: spheres must not intersect with other spheres must completely locate inside the volume, and the volume covered by spheres is maximized using greedy strategy by subtracting the volume of the largest sphere for every iteration where largest sphere is found using a distance transformation. In our technique, sphere placements are no longer on the skeleton line. Instead, the spheres are placed wherever the maximum distance value occurred inside the ROI during that iteration (Fig. 2). We applied our maximum distance sphere packing strategy algorithm successfully on many MRIs using the Slicer3D platform; a new module in Slicer3D to be used for different shape approximation purposes. The spheres centers of the 3D object represent a spatial template as a graph. The graph is a representation of the intersection of the sphere's centers that represent vertices of the graph of all maximum distances contained inside the 3D object, and edges connected each two consecutive generated spheres (Fig. 3). Ordering of the spheres is important for example: B, C, A will give different signature graph than A, B, C. We introduce a measure called epsilon-rotation invariant. Such geometric accuracy of MRI is practical especially when it used for planning radio surgery. Testing different angles of the image for beam planning is needed. Rotating the 3D volume must give the similar arrangement of sphere packing. We captured inner distances between two consecutive spheres' centers of our shape descriptors as an approximation to compute the difference between the two 3D shape descriptors, before and after the rotation. This graph distances representation is useful to abstract a geometric meaning of the 3D shape and to characterize the connectivity information. From Figure 2, assuming we rotated a 3D tumor, apart from how close d1 is to d' 1 and d2 is to d' 2 etc., we also look at the inner spheres' centers distances between center of the original spheres compared with the corresponding distances on the rotated volume by finding the ratio as follow: The inner distances between the spheres capture the distances before and after the rotation of 3D object, and find the sphere packing descriptors. In other words, we find how similar the spherical coverage is before and after, and intuitively compare that to the graph inside the spheres. Although we did not implement the orientation of such inter-distances, we expect that to be closely related for better for our distance transformation based shape descriptors. Intuitive idea is that apart from radius being equal, the relationship between the centers should also be similar between one sphere to another. Our algorithm descriptor map entries correspond to the Euclidean distance between spheres' centers and these values are arranged in a manner that preserve the relative position of each sphere. We implemented our method in Slicer3D (Fig. 4). The Slicer3D , is an open source medical visualization tool. The 3D slicer builds on top of different libraries such as VTK, ITK, CMake, NA-MIC, Qt and Python . Also, it contains more than a hundred modules written in C++ or Python to provide researchers many common tools and rich implementations to achieve and implement their goals. The visualization toolkit (VTK) framework is an open source with C++ libraries that contains many filters for data representation/visualization. We developed our Slicer3D Python module for sphere packing to work with the VTK for volume rotation. 3D arbitrary Original Rotated rotations are introduced for medical images as an extension of our previous work carried out for sphere packing . We used VtkTransform to apply rotation via 4x4 matrix multiplications. Our algorithm rotates images any number of degrees around x, y, and z axes. Any arbitrary rotation can be described by specifying the coordinates of the object in 3D space and rotation angels. Unlike 2D rotation, 3D rotation occurs along an arbitrary axis. Suppose the rotation angle is a, the rotations about three major axes uses well known formulae: Markup Language (MRML) and display images in physical space using patient coordinates system RAS (Right Anterior Superior), based on the information of the image spacing, origin, and direction. When applying rotation, we used the spheres packing information along with the origin and spacing. Thus, before we apply the rotation, we need to know the data of the volume: -Position: the 3D coordinates of the object. -Bound: the bound box of the object represented as (xmin, xmax, ymin, ymax, zmin, zmax). -Origin: it is the position of the first voxel in the patient coordinate. It is the space origin, which is the center of all rotations -Spacing: it is the voxels distances along each axis in the image. Applying rotation using VtkTransform is done by following six phases as follow: • Phase 1: Crate and add a transformation node. We first create a TransformNode using VtkMRMLTransformNode, then add that node to the MRML scene. This node contains the transform ID and can store any linear transformations of composite of multiple transformations. RotateX, RotateY, and RotateZ create the rotation matrix. Since VtkTransform rotate the object around the origin, the rotation algorithm performs the following steps to rotate the volume about its center. The volume is first translated to its center so that its centroids lie on the center of the image instead of the origin. The ing volume is then rotated according to transformation chosen by the user (x, y, and z angles values). Then, translate the volume back to its original pose. • Phase 4: Apply transformation. Where tNode is the transform node. GetMatrix is used to return the current values to be used for the view manipulations such as rotate the current values in x, y, z angles. So, the current values (vtkMatrix 4x4) are multiplied by transformation matrix. Applying transformation is basically done by multiplying current node with the transform node and stored in simple linear transform: VtkMRMLVolumeNode(tNode) * transformNode • Phase 5: Concatenate multiple (nested) transformations and attach volume to transform node. OutputVolume. SetAndObserveTransformNodeID(tNode. GetID) • Phase 6: Harden transform. Describe applying transformations and save it as a transformed model. Invoking transform model and harden transform the volume to get the correct new orientation, which will be stored in the image header. Thus, harden transformation is used for: changing orientation and generation of the output volume. Our heuristic is based on the greedy concept of generating largest spheres first, is O(n*K) where n is the number of spheres found to satisfy the chosen coverage and K is per sphere iterative constant where K= number of voxels in the volume data set. The pseudo code for Euclidean sphere packing rotation algorithm is as follow: Experimental demonstrate the effectiveness and efficiency of the proposed method. In our experiments, we used thirty MRIs of segmented brain tumors from the BRATS dataset (Menze, Bjoern H and) separated on three datasets with ten patients on each (Fig. 5). The three datasets are manually revised and delineated by experts broadcertified neuroradiologists and radically different in size, shape and complexity. The tumor sizes range from 248318 to 12948 pixels. The proposed method is applicable to different volume shape representations and different arbitrary rotations angles (Fig. 6). Our attempt for seeking a rotation invariant descriptor is to improve the matching similarity performance inspired by region's partition by non-intersecting spheres packing and based on maximum distance. We first divide the volume with spheres. Then, distances ratios and radii of each sphere are calculated to be compared with the correspondence on the rotated volume of distance ratio and radii values (Fig. 1). More similarity measurements are investigated such calculating the accuracy along with the Mean Absolute Error (MAE) of our algorithm. We developed an epsilon value measure for similarity based on our study. This allowed us to manage differences that due to the fact that voxel sizes also change and are within epsilon (e) of each other. In our study, we observed interesting patterns looking at the radius of spheres, and they are being close enough before and after the rotation. The spheres radiuses and distance ratios are actually within epsilon (e) value criteria. The Epsilon is the maximum distance in terms of voxel size and is always given within a small range of numbers. Thus, after analyzing our 30 patients MRIs, the epsilon values of the difference spheres' radius before and after rotation are under one unit of difference, specifically within the value of 0.8 mm. Therefore, any radiuses within ± 0.8 are meant to be acceptable and there is then a high probability that the 3D volumes are similar when there are no multiple spheres with the same sphere-radius. Since previous epsilon value is based on the 50% of the packing density, we tested our epsilon value under different packing densities such as 60%, 70% and 90%. We found that, our epsilon value is consistent under ± 0.8 (Fig. 7). On the other hand, when we consider the ratio of distance between two consecutive spheres in both before and after sphere packing list, the epsilon value between the distance ratios of the original volume and the rotated volume is within ± 2.5. That means, the difference in distance ratio between any consecutive spheres has to be within ± 2.5. However, increasing the packing density strongly increase this value to ± 4 in 60%, ± 12 in 70%, and ± 32 in 80% of packing density (Fig. 8). This is as expected (discussed in the next section) as we go deeper in the list of spheres. The accuracy of our technique to approximate the same radiuses/ratio values after rotation is calculated for each patient data, we divided each radius or ratio in the original volume by its corresponding radius/ratio after rotation to see how well our algorithm works. The overall average radiuses accuracy percentage is 96.86% (Fig. 9). On the other hand, differences between the ratio of distance before and after, the overall Ratio average accuracy of getting same ratios is 69.23% (Fig. 10). We noted that our are driven by a good accuracy and further 3D-spatial improvements will fetch better . To increase the accuracy, we further tested the accuracy of some of our by varying the volume dimension (voxel size). Our datasets patients' grid size varies so we increased the dimensions to different values to have bigger grid size with a greater number of voxels of smaller sizes. We find that the decrease in voxel size change the accuracy level. The more the voxel size decreased (grid became finer), the accuracy increased (Fig. 11). This is expected because smaller voxel sized provide more accuracy than the bigger size Still, we analyzed the data further. For each patient, we computed the absolute error between the original and the rotated radiuses/ratios. Absolute values were estimated across a range of different patients. Absolute error = (Before_radius -After_raduis). Then, the mean absolute error (MAE) was calculated for patients using the distribution of ratios and radiuses in 30 patients. The closer this value to the zero indicate 1 17 33 49 65 81 97 113 129 145 161 177 193 209 225 241 257 273 289 305 321 337 Patient the great algorithm approximation to cover the targeted object. The overall MAE of our algorithm is 0.2. In medical applications , we believe that this measure could be important because the absolute error represents the risk of developing recurrent disease because this value indicates the untreated cells/voxels. Being able to differentiate between patient with highest and lowest absolute risk of recurrence is an important task in order to diagnose the patient with the appropriate treatment. Therefore, the MAE play an important role to differentiate whom radiotherapy can yield to meaningful benefits. The spheres radius works the best for our study for finding the similarity after rotation. Even though there are differences between the total number of calculated distances before and after rotation, our algorithm accuracy is reasonably high because it is able to calculate almost similar radiuses each time within epsilon. The consistency of spheres radius is because our algorithm at each iteration finds the maximum radius distance to pick first, so increasing the number of packed spheres to cover the required voxels based on the desired packing density doesn't affect the epsilon value. Changing the topology due to equal spheres is the main reason of the increase of the epsilon value of the distance ratios. The algorithm decision of choosing which sphere of the same size to place first, is the big issue here. Therefore, increasing the number of packed spheres will significantly increase the changes in topology which will in increasing the epsilon value. When the radiuses are equal, the descriptor graph before and after might change considerably based on which sphere our algorithm suggests. In this case, aggregate we will need to collect all those spheres which are equal and replace them by the average of the center of the sphere in the shape descriptor, and so then (e) value will be similar in the shape description before and after the rotation. Thus, set of spheres whose radius are equal are replaced with one sphere. That is expected to reduce the epsilon (e) value further. Moreover, the topology changing in our study affect our accuracy . We believe that eliminating equal spheres by using the enclosing sphere in our implementation will decrease the distance ratio comparing the shape descriptors before and after the rotation of the ROI. Our novel medical visualization techniques promise to improve the efficiency, diagnostic quality and the treatment. The field of 3D shape approximation and similarity have been a focus in the area of geometry in general for several hundred years now. Shape analysis for feature extraction is the key problem in the shape approximation and similarity issues. The best way for similarity matching is to identify certain shape signatures (prominent features in the image). These signatures are then compared between the transformed images through similarity assessment, distance computation or any other appropriate methods. This paper presented a method for defining a possible invariant shape descriptor from 3D-image or 3D-volume data to be used to match the objects from different rotations/viewpoints. Our method can be applied to a wide variety of data types such as 2D images and even polygonal meshes. Our heuristics is e-invariant and has an impressive of 96% invariant under rotations. The experimental prove the effectiveness of our novel idea. The proposed system was fully software implemented in Slicer3D and has been tested on 30 patient's databases. For future works, we will apply other measures such as 3D-spatial sorting based on the spheres found, or identifying a minimal volume enclosing sphere surrounding all spheres of equal radius (as mentioned earlier) to improve epsilon (e) value further. Moreover, as Slicer3D is experimental, not FDA approved, yet used worldwide, our plan is to upload our implementations under BSD license so that worldwide communities can try the system and provide more feedback using their 3D volume data and reporting e-value for their data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Packing region of Interest (ROI) such as cancerous regions identified in 3D Volume Data, Packing spheres inside the ROI, rotating the ROI , measures of difference in sphere packing before and after the rotation.

A novel approach that detects outliers from image data,  while preserving the classification accuracy of image classification