Question: Represent the scientific article to find a short TLDR

Question: In spite of their great success, traditional factorization algorithms typically do not support features (e.g., Matrix Factorization), or their complexity scales quadratically with the number of features (e.g, Factorization Machine). On the other hand, neural methods allow large feature sets, but are often designed for a specific application. We propose novel deep factorization methods that allow efficient and flexible feature representation. For example, we enable describing items with natural language with complexity linear to the vocabulary size—this enables prediction for unseen items and avoids the cold start problem. We show that our architecture can generalize some previously published single-purpose neural architectures. Our experiments suggest improved training times and accuracy compared to shallow methods. In recent years, predictive tasks that traditionally have been solved with factorization are now being studied within the context of neural networks. These solutions often work as black boxes, and many times they are designed specifically for a single task with an arbitrary network that may not have much justification. We propose Deep Structured Factorization Machine, a family of general-purpose factorization techniques that can be used stand-alone or as a "design pattern" within a larger neural network. Our work provides some insight into how to enable general-purpose factorization within neural architectures without losing interpretability and a principled design. Previous factorization methods do not scale to large feature sets and make strong assumptions about their latent structure. Our main contribution is that we enable a general-purpose framework that enables efficient factorization of datasets with complex feature sets. For example, applications of factorization in natural language scale quadratically in the number of words in the vocabulary. Our solution allows inference with linear runtime complexity on the vocabulary size. Previous work has explored how to improve factorization's accuracy (see § 3.3) with its current limitations withstanding; alternatively, some have proposed how to make it tractable for a particular domain-for example, text BID22. We believe that we are the first ones to propose an efficient general-purpose method. Interestingly, our experiments indicate that Structured Deep Factorization has large improvements in predictive accuracy and runtime compared to some recent ad-hoc models. Factorization Machine BID15 ) is one of the most succesful methods for general purpose factorization. BID15 formulated it as an extension to polynomial regression. Consider a degree-2 polynomial (quadratic) regression, where we want to predict a target variable y from a feature vector x ∈ R n:ŷ DISPLAYFORM0 Here, λ 1 and λ p are one-way and pairwise interactions: DISPLAYFORM1 In words, n is the total number of features, the term b 0 is an intercept, b i is the strength of the i-th feature, and w i,j is the interaction coefficient between the i-th and j-th feature. The function ω is an activation. Choices for ω include a linear link (ω(x) = x) for continuous outputs, or logistic link (ω(x) = log exp(x) exp(x+1) ) for binary outputs. Factorization Machine replaces the two-way individual pairwise parameters with shared parameters β i. This is a rank-r vector of factors-embeddings in the neural literature-that encode the interaction between features: DISPLAYFORM2 Intuitively, the dot product (·) returns a scalar that measures the (dis)similarity between the two factors. Polynomial regression has n 2 interaction parameters, and Factorization Machine has n × r. While using r n makes the model less expressive, this is often desirable because factorization is typically used when features have some shared latent structure. Factorization Machine may dramatically reduce the number of parameters to estimate; however, inference has runtime that scales quadratically with the number of features-it requires O(n 2) dot products. BID15 shows that when the feature vector x consists only of two categorical features in one-hot encoding,Factorization Machine is equivalent to the popular Matrix Factorization algorithm BID9 ). Factorization Machine has the following working-assumptions and limitations:1. Strong parameter sharing. Factorization Machine assumes that all of the pairwise interactions factorize into a common shared space. This may not be the case when using large and complex feature sets. To the extent of our knowledge, softening this assumption has not been studied in the literature before. We study how to do so in § 3.1. 2. Intractable for large feature sets. Doing inference on Factorization Machine has a runtime complexity that scales quadratically with the number of features. Thus, it is intractable for complex feature sets such as text. We address this in § 3.2.We propose Structured Factorization Machine as a building block that we will extend with neural methods. It is a simple yet effective way to model structure between groups of features. In Factorization Machine, all the features interact with each other. In the structured model, the features only interact with features from a different group. We do this by defining groups of features κ. We define Structured Factorization Machine as: DISPLAYFORM0 The interactions occur from features in different groups: DISPLAYFORM1 For example, consider a model with four features (n = 4). If we define κ = {{1, 2}, {3, 4}}, feature x 1 would only interact with x 3 and x 4. Without loss of generality, we could define a model that is equivalent to a shallow Factorization Machine by allowing each feature to be in a singleton group: κ = {{1}, {2}, {3}, {4}}.Figure 1 compares existing factorization methods with our novel models. In the rest of this section we review them. DISPLAYFORM2 We now contrast Polynomial Regression (Equation 1) and Factorization Machine. Polynomial regression is convex, and thus can be optimized globally. Because Factorization Machine is nonconvex, optimization can only guarantee to find local optima. The factorization assumption is more desirable than Polynomial Regression when there is shared structure between the features. The larger number of parameters of Polynomial Regression makes it likelier to overfit. However, we consider two properties of independent interactions that are desirable. First, it is unclear if all interactions strongly factorize when using very large feature sets. For example, some interactions may have different importance-perhaps because some features are irrelevant and should be discarded. Second, it may be easier to optimize a model with individual parameters than a model with factorized parameters. Polynomial regression converges quickly because the likelihood function is convex-a unique solution exists. We aim to combine the advantage of both models. We propose a hybrid factorized-independent method that we call Structured Deep-Out Factorization Machine, because it extends Structured Factorization Machine outside the dot product (see FIG0 : DISPLAYFORM0 Here, the the embeddings β are factorized and shared across the pairwise interactions. The parameters w are independent to each interaction group defined by κ. The function ψ is any activation function. If we were interested in interpreting the parameters, we would constrain w to be non-negative and choose ψ to be linear. Alternatively, the Rectified Linear Unit (ReLU) activation may be desirable because it is easy to optimize BID10: DISPLAYFORM1 We leave for future work the consideration of deeper networks. Structured Deep-In Factorization Machine allows treating a group of features as a single one; it extracts features from each feature group, and builds latent factors from them. Consider FIG0: the first group only has a single feature which is projected to an embedding (just like a regular Factorization Machine); the second group has multiple features, which are together projected to a single embedding. More formally: DISPLAYFORM0 In this notation, x κi is a subvector that contains all of the features that belong to the group κ i. Thus, DISPLAYFORM1 The intuition is that by grouping (sub-)features as a single entity, we can can reason on a higher level of abstraction. Instead of individual sub-features interacting among each other, the entities interact with other entities. Here, φ i is a feature extraction feature that inputs the i-th feature group of the instance, and returns an embedding. The simplest implementation for φ i is a linear fully-connected layer, where the output of the r-th entry is: DISPLAYFORM2 Within a group, a feature extraction function may allow for subfeatures to interact with each other. Across groups, entities interact with each other via the output of φ only. Leveraging item descriptions to circumvent the cold-start problem We now describe how we can use Structured Deep-In Factorization Machine for large feature sets-such as natural language. In this scenario Factorization Machine is not tractable: if we use each word in the vocabulary as independent feature, we have to consider the interaction of each word with every word in the vocabulary and the other features, which means the number of interaction terms would depend quadratically on the size of the vocabulary. A traditional work-around is to ignore the features, and use matrix factorization by using a unique one-hot index for each item (e.g, each image/text), which avoids the quadratic dependence on the vocabulary size, but in this case the model suffers from the cold-start issue: inference is not possible for a new item not seen during training, even if it shares many words with existing items. We can use the Deep-In model to both use large feature sets and overcome the cold-start problem. This is only possible when there is an alternative description of the item available (for example an image or a text). In FIG1, we show how we address this problem by treating the words as indexed features, but placed within a structured feature group κ w. A feature extraction function φ acts on the features in κ w, and the other features interact with the words only via the output of φ. Notice that this implies we can precompute and store the latent factors of the labels seen during training, and predictions during inference can be sped-up. For example if we have two feature groups (e.g, a label and an item), first we compute the feature extraction function to the unseen items and their embeddings, and then we simply apply a dot product over the stored vectors of the labels. FIG0 shows a Neural Factorization model that has been discovered multiple times BID4 BID5. It replaces the dot product of factors with a learned neural function, which has been shown to improve predictive accuracy for various tasks. Unlike our work, this method has the same drawbacks as regular Factorization Machine for feature types such as natural language. As discussed in §3.2, applying this method for text-based inputs would in either the cold-start problem or quadratic dependence on the vocabulary. It would be straightforward to combine Neural Factorization with Structured Deep-In Factorization Machine, which is not explored in this work. Note that if the dot product is replaced with a neural function, fast inference for coldstart documents using pre-computed label embeddings is no longer possible. DISPLAYFORM3 We can learn the the parameters of a deep factorization model θ using training data by minimizing a loss function L: DISPLAYFORM0 Here, y(x) is the true target value for x obtained from training data, andŷ(x) is the one estimated by the model; the hyperparameter γ controls the amount of regularization. For the labeling and classification tasks, we optimize the binary cross-entropy for y ∈ {0, 1}: DISPLAYFORM1 For the regression tasks where the target value is continuous, we optimize the mean squared error (MSE): DISPLAYFORM2 Neural models are typically learned using mini-batch updates, where the incremental descent is performed with respect to several instances at a time. For the implementation of this paper, we built our models using the Keras programming toolkit BID2, that is now part of Tensorflow BID0. It enables automatic differentiation, and is bundled with a generalpurpose optimization algorithm called ADAM BID8 ) that needs no tuning of gradient step sizes. For the multi-label classification tasks, we use Structured Factorization Machine with a binary output. In this case we would have at least two feature groups-one of the feature groups is the label that we want to predict, and the other group(s) is the input from which we want to make the prediction. The output indicates whether the label is associated with the input (y = +1), or not (y = 0). The datasets we use for our labeling experiments only contains positive labels, thus for each training example we sample a set of negative labels equal to the number of positive labels. We choose one of the following sampling strategies according to the best validation error, in each case excluding the actual positive labels for each training example -(i) uniformly from all possible labels, or (ii) from the empirical distributions of positive labels. Other sampling strategies have been proposed BID17 BID16; Anonymous, In submission). We now address our working hypotheses. For all our experiments we define a development set and a single test set which is 10% of the dataset, and a part of the development set is used for early stopping or validating hyper-parameters. Since these datasets are large and require significant time to train on an Nvidia K80 GPU cluster, we report on only a single training-test split. For the labeling and classification tasks we use evaluation metric called Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC). Since we only observe positive labels, for such tasks in the test set we sample a labels according to the label frequency. This ensures that if a model merely predicts the labels according to their popularity, it would have an AUC of 0.5. A caveat of our evaluation strategy is that we could be underestimating the performance of our models-there is a small probability that the sampled negatives labels are false negatives. However, since we apply the evaluation strategy consistently across our methods and baselines, the relative difference of the AUC is meaningful. We choose the AUC as a metric because it is popular for both classification and ranking problems. For the regression tasks, we use MSE as the evaluation metric. In preliminary experiments we noticed that regularization slows down convergence with no gains in prediction accuracy, so we avoid overfitting only by using early stopping. We share most of the code for the experiments online 1 for reproducibility. Factorization algorithms are desirable for smaller feature sets (e.g, users and items) that have shared latent structure, because a more parsimonious model is less prone to overfitting. We now investigate if relaxing the factorization constraints for models with larger and more heterogeneous features is useful. We do this by comparing deep-out and a shallow structured models under the same conditions. For reproducibility details on our grid-search see Appendix A.1. We test this on classification, regression, and collaborative filtering (with implicit feedback prediction) tasks, respectively:• Subscription prediction We predict the binary outcome of whether a marketing campaign is successful using the UCI Bank Marketing Data Set BID12 BID14. This is a small dataset with only 45,211 observations. We use 17 categorical and real-valued features.• Airline delays We predict the real-valued delay of american flights using the 2008 RITA dataset 2. using only 8 categorical features. This dataset is 7 million rows.• Course prediction On this collaborative filtering task with implicit feedback, we use partial study plans collected from a company that has an educational-technology presence. We only consider data from students with three or more course enrollments, and courses that have had at least enrollments from 10 students. The ing dataset contains approximately 36 million enrollment instances from roughly 4.7 million students and 2,578 courses (out of 2,930 that exist on the ontology). For sampling implicit feedback, we sample the course and subject variables together. The features we use are all categorical: student id, course id, subject level 1, subject level 2, subject level 3, university id, student year (discretized), and data source types. For both shallow and deep models, we define the groups the same way: we group each continuous feature by itself; we group the multiple dimensions of the one-hot encoding of categorical variables together. This way, for example, the 12 dimensions that are used to encode a "month" feature don't interact with each other. TAB0 summarizes our . Adding a handful of deep-out parameters is very effective for improving forecasting quality or training time. For example, shallow factorization does not do much better than random chance in the subscription prediction, but the deep-out approach improves the AUC by roughly 35%. We hypothesize that this dataset has features with little shared structured. On the courses dataset, the deep method is almost twice as fast as the shallow method, with some modest improvement on performance. In this section, we focus on natural language processing tasks. TAB1 provides descriptive statistics of the datasets that we consider. To choose the feature extraction function φ for text, we use a Convolutional Neural Network (CNN) that has been shown to be effective for natural language tasks BID7 BID20. In Appendix A.2 we describe this network and its hyper-parameters. Instead of tuning the hyper-parameters, we follow previously published guidelines BID21. We experiment on two datasets to predict a label from text, obtained from an educational technology company: 2. Skill labeling. This dataset contains job posting texts annotated with the skills that are deemed to be useful by hiring managers. Thus, we try to predict the skills required for a job, particularly those that are not explicitly mentioned in the job posting or description. For each task, we build two feature groups: the labels and the document text. The document texts are represented as a sequence of one-hot encoded words. As we describe in § 3.2, it is reasonable to use Matrix Factorization and ignore the item descriptions. TAB2 shows that Structured Deep-In Factorization Machine outperforms this strategy. Unfortunately, we cannot compare to Factorization Machine because it is intractable with a large vocabulary, as it would lead to interactions that scale quadratically in the vocabulary size (100,000 in our case). Matrix factorization cannot generalize to unseen items. Instead, we compare to Collaborative Topic Regression (CTR-Wang & Blei FORMULA0 Because the implementation we found is relatively slow, we cannot test them on the concept and skill datasets. Instead, we use the CiteULike dataset which consists of pairs of scientific articles and the users who have added them to their personal libraries. We use it to predict users who may have added a given article to their library. We compare the performance of Structured DeepIn Factorization Machine with CTR using pre-defined cross-validation splits 4 . We use 1% of the training set for early stopping. For CTR we use the hyper-parameters reported by the authors as best, except for r which we found had a significant impact on training time . We only consider r ∈ {5, 10, 15} and choose the value which gives the best performance for CTR (details in Appendix 6). On the warm-start condition, CTR has an AUC of 0.9356; however, it shows significant degradation in performance for unseen documents an it only performs slightly better than random chance with an AUC of 0.5047. On the other hand, Deep-In FM achieves AUC of 0.9401 on the warm-start condition, and it only degrades to 0.9124 on unseen documents. For completeness, we also tested Deep-In FM on cold-start prediction for the Concepts and Skills dataset, on 1% of documents unseen during training. We obtained AUC of 0.92 and 0.67, respectively, which are within 3% of the warm-start . Deep-In FM can also be trained over ten times faster, since it can leverage GPUs. 5 We also note that we have not tuned the architecture or hyper-parameters of the feature extraction function φ for each dataset and greater improvements are possible by optimizing them. We now compare with a method called DeepCoNN, a deep network specifically designed for incorporating text into matrix factorization BID22 -which reportedly, is the state of the art for predicting customer ratings when textual reviews are available. For Deep-In FM we use the same feature extraction function (see Appendix A.3 for details) used by DeepCoNN. We evaluate on the Yelp dataset 6, which consists of 4.7 million reviews of restaurants. For each user-item pair, DeepCoNN concatenates the text from all reviews for that item and all reviews by that user. The concatenated text is fed into a feature extraction function followed by a factorization machine. In contrast, for Structured Factorization, we build 3 feature groups: item identifiers (in this case, restaurants), users and review text. TAB3 compares our methods to DeepConn's published because a public implementation is not available. We see that Structured Deep-In Factorization Machine provides a large performance increase when comparing the reported improvement, over Matrix Factorization, of the mean squared error. Our approach is more general, and we claim that it is also more efficient. Since DeepCoNN concatenates text, when the average reviews per user isn u and reviews per item isn i, each text is duplicated on averagen i ×n u times per training epoch. In contrast, for Deep-In FM each review is seen only once per epoch. Thus it can be 1-2 orders of magnitude more efficient for datasets wherē n i ×n u is large. We present a general purpose method for factorizing large feature sets; we demonstrate it in several applications, such as using text to enable prediction for unseen items and circumvent the cold-start problem. Future work may soften our requirement of domain knowledge-in general, our methods require feature groups and feature extraction functions defined by experts. We did not pursue an exhaustive comparison with previously published methods; for example, there are other algorithms that rely on Bayesian optimization BID3 to infer the item embeddings from text which we did not benchmark. Although we apply our methods on six datasets altogether, further experimentation may be able to situate under which conditions our methods are effective. Our methods generalize previously published single-purpose neural networks. For example, TagSpace BID20 ) is a very successful method, but it is limited to a single textual feature. With the correct feature extraction function, Structured Deep-In Factorization Machine can be used to implement a TagSpace model. Compared to previous general-purpose approaches, our work makes less assumptions about the training data and allows more flexibility. We provide evidence that the factorization hypothesis may be too restrictive-when relaxed we see higher predictive accuracy with a dramatic improvement of training speed. We show experimental outperforming an algorithm specifically designed for text-even when using the same feature extraction CNN. This suggests that the need for ad-hoc networks should be situated in relationship to the improvements over a general-purpose method. To the extent of our knowledge, our work is the first to propose a general purpose factorization algorithm that enables efficient inference on arbitrary feature sets. For the courses dataset, our experimentation strategy killed experiments that took too long. In practice, that means that some shallow experiments with batch size of 5,000 observations were not executed. Here we describe the details of the feature extraction function φ used in our experiments for labelling in §4.2.1 and §4.2.2. An overview of the network is given in FIG3. We choose the most popular words of each dataset to build a vocabulary of size n, and convert the words of each document to a sequence of length t of one-hot encodings of the input words. If the input text is shorter than t, then we pad it with zeros; if the text is longer, we truncate it by discarding the trailing words. Therefore, for a vocabulary size n, the input has dimensions t × n, and this matrix is then passed through the following layers:1. We use an embedding layer to assign a d-dimensional vector to each word in the input passage of text. This is done through a d × n-dimensional lookup table, which in an t × d matrix.2. We extract features from the embeddings with functions called convolutional filters (also called feature maps). A convolutional filter is simply a matrix learned from an input. We learn f filters that are applied on groups of m adjacent word embeddings, thus each of our filters is a d × m matrix of learned parameters. Filters are applied by computing the element-wise dot product of the filter along a sliding window of the entire input. The ing output for each filter is a vector of length t − m + 1. We also apply a ReLU activation to the output of each filter. 3. Consider the case of inputs of different lengths. For very short texts, the output of the filters will be mostly zero since the input is zero-padded. To enforce learning from the features of the text, and not just its length we apply a function called 1-max pooling to the output of the filters: from the t − m + 1 output vector of each filter, we select the maximum value. This yields a vector of length F, a representation of the passage which is independent of its length.4. We learn higher-level features from the convolutional filters. For this, we use a fully connected layer with p units and a ReLU activation, 5. During training (not in inference), we prevent the units from co-adapting too much with a dropout layer BID18. Dropout is a form of regularization that for each mini-batch randomly drops a specified percentage of units.6. the final embedding for x j (that is used in the factorization) is computed by a dense layer with r output units and an activation function, where r is the embedding size of our indexable items. We set the maximum vocabulary size n to 100,000 words, and input embedding size d to 50 for all experiments. We initialize the input word embeddings and the label embeddings using Word2Vec BID13 We have have not evaluated multiple architectures or hyperparameter settings and obtain good on diverse datasets with the same architecture, which was designed followed recommendations from a large scale evaluation of CNN hyper parameters BID21. We set the number of convolutional filters f to 1,000, and the dropout rate to 0.1. The maximum sequence length t was chosen according to the typical document length (5,000 words for the concept dataset, 400 for the skills and 350 for CiteULike), the embedding size was set to r = 100 for concepts, and r = 200 for skills. For the CTR dataset since we use very small values of r, due to the tendency of the ReLU units to'die' during training (output zero for all examples), which can have a significant impact, we used instead PReLU activations BID6 for the final layer, since they do not suffer from this issue. The CNN architecture used for DeepCoNN BID22 ) is similar to that described above in Appenix A.2. It consists of a word embedding lookup table, convolutional layer, 1-max pooling and a fully connected layer. We use the hyper-parameters that the authors report as best -100 convolution filters and 50 units for the fully connected layer. We set the word embedding size to 100, the vocabulary size to 100,000 and the maximum document length to 250. To compare Deep-In FM with Collaborative Topic Regression, we choose the embedding size r ∈ {5, 10, 15} for which CTR performs best. The other parameters were set to those reported by the authors as best. The are shown in TAB5.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Scalable general-purpose factorization algorithm-- also helps to circumvent cold start problem.


Question: represent the scientific article to find a short tldr

Question: Augmented Reality (AR) can assist with physical tasks such as object assembly through the use of “situated instructions”. These instructions can be in the form of videos, pictures, text or guiding animations, where the most helpful media among these is highly dependent on both the user and the nature of the task. Our work supports the authoring of AR tutorials for assembly tasks with little overhead beyond simply performing the task itself. The presented system, AuthAR reduces the time and effort required to build interactive AR tutorials by automatically generating key components of the AR tutorial while the author is assembling the physical pieces. Further, the system guides authors through the process of adding videos, pictures, text and animations to the tutorial. This concurrent assembly and tutorial generation approach allows for authoring of portable tutorials that fit the preferences of different end users. Physical task guidance can be delivered via Augmented Reality (AR) since assembly often requires both hands and continuous attention to the task. Additionally, assembly tutorials have instructions directly associated with physical objects, so AR can reduce the need for excessive context switching between the instructions and the physical structure by projecting those instructions into the environment. These benefits have been demonstrated in fields such as Facilities Management, Maintenance, and Internet of Things (IoT) device management. Additionally, prior work in AR assembly guidance has shown that these benefits can translate to carrying out assembly tasks. While significant previous work has looked at the benefits of following tutorials in AR, much less has looked at how to author these tutorials. Beyond the technical requirements of an authoring interface, an ideal tutorial may look different depending on the end user of the tutorial. This problem is exacerbated in AR as there are many different modalities in which tutorial content can be presented. While one person may appreciate guiding animations in AR, another may prefer static text and images, and yet another may prefer video tutorials from one or multiple perspectives. With AuthAR, we present a system for building tutorials for assembly tasks that can accommodate the needs of these different types of end users. AuthAR generates video, and pictorial representations semi-automatically while the tutorial author completes the task. Furthermore, AuthAR allows tutorial authors to create and refine a tutorial in situ, integrating content authoring into the process of completing the task. This approach adds little additional overhead and reduces the need for post-processing of the tutorial. This paper presents the AuthAR system for generating mixed media assembly tutorials. Informed by prior work on content/tutorial authoring, and tutorial playback and walkthrough, we build the system with an eye toward non-obtrusive content authoring and generation of important components for tutorial playback, summarized in a set of design guidelines. We validate the system's ability to create a tutorial by stepping through the process of creating a tutorial to build a laptop stand, automatically generating an XML representation of the tutorial. Initial observations suggest the tool will be valuable, and possible ways the system could be extended and refined in future iterations. AuthAR builds on prior research in the areas of AR tutorials and content authoring, as well as principles of mixed media tutorial design. AR is often applied to assembly tasks for its ability to project instructions into the environment such that they are spatially relevant. Radkowski et al. demonstrate assembly of a 3D puzzle and provide evidence that AR supports faster assembly than traditional methods. Similarly, Henderson et al. demonstrate the use of projected guidance for engine assembly. Prior work on AR guidance design has shown that abstract representations (3D text, arrows) can be more effective for complex tasks than a more concrete representation (virtual models of the pieces and fasteners), which is sensible for simpler tasks. In that work, the authors found information-rich 2D representations to be more effective than either of the AR representations in some cases. One theory is that AR is only justified when the task is sufficiently difficult, such that the time to process the information is insignificant compared to the time to perform the task. So even for physical tasks in which instructions' spatial relevance could be increased by projecting into the environment, tutorials should provide users the ability to view the step(s) in a more familiar picture/video format when needed or preferred. The need for these mixed media tutorials is apparent, however little work has explored authoring of these tutorials for physical tasks. Outside of the realm of tutorial authoring, numerous systems have explored content creation in augmented reality to abstract low-level programming from the creator, in turn lowering the threshold for participation with AR. Many AR content authoring systems give users a collection of 3D models to place and manipulate in an environment or to overlay on a video stream, allowing users to create AR content without programming expertise generally required to build such scenes. Other AR content creation systems target specific end users for participation by domain experts in areas such as museum exhibition curation tour guidance, and assembly/maintenance. Mota et al. provides a survey of existing AR content creation tools, classifying as standalone applications vs. plug-ins to existing applications and platform-specific vs. platform-independent. Of particular interest to us are authoring tools that allow for creation of training experiences. Built on Amire's component-based framework, Zauner et al. presents a tool to enable authoring of assembly task guides in augmented reality. With this tool, the author puts visually tracked pieces together hierarchically to create an assembly workflow assistant in AR. Alternatively, the expert can collaborate remotely, rather than creating a training experience ahead of time. In this scenario, the expert can annotate the live video feed provided by the trainee's AR headset for varying levels of guidance on-the-fly. These systems require explicit enumeration of every component to be added to the created scene whereas our system generates content semi-automatically (segmenting video, recording changes to transform and detecting use of a tool) where possible. Moreover, our system only requires this manual input for augmentation and refinement of the tutorial while the bulk of authoring is done in situ. Within the domain of software tutorials, Chi et al. provides design guidelines for mixed media tutorial authoring with their MixT system for software tutorials. They list scannable steps, legible videos, visualized mouse movement and giving control to the user on which format to view as important components of a mixed media tutorial. Carter et al. echos this sentiment, with their ShowHow system for building a tutorial of videos and pictures taken from an HMD noting that mixing different media is important in lieu of relying on a single media type. Our work builds upon these concepts of mixed media tutorials but applies them to AR authoring of physical task tutorials. The rest of this subsection discusses the use of three popular media used for both software and physical task tutorials: videos, images, and interactive guidance. Prior work on video-based tutorials has applied different strategies to the challenges of video segmentation and multiple perspectives. DemoCut allows for semi-automatic video segmentation such that these demonstration videos are appropriately concise without requiring significant postprocessing. Chronicle allows for video tutorials based on the working history of a file. As the file changes, their system generates a video tutorial of how the file changed. For physical tasks, Nakae et al. propose use of multiple video perspectives (1 st person, 3 rd person, overhead) to record fabrication demonstrations and semi-automatic generation of these videos. Prior work also uses captioned and augmented images for physical task and software tutorials. Image-based tutorials can be generated automatically from demonstration as is done with TutorialPlan, a system that builds tutorials to help novices learn AutoCAD. Images can also represent groups of significant manipulations (such as multiple changes to the saturation parameter) as Grabler et al. demonstrated for GIMP tutorials. For physical tasks, prior work explored use of AR to retarget 2D technical documentation onto the object itself, providing spatially relevant augmentations. Interactive tutorials guiding users where to click, place objects, and even move their hands have become increasingly popular. For example, EverTutor automatically generates tutorials for smart phone tasks such as setting a repeating alarm or changing font size based on touch events. They found improved performance and preference toward these tutorials over text, video, and image tutorials. In the realm of physical assembly tutorials, use of visual and/or depth tracking allows for automatic generation of tutorials based on changes to location and rotation. Further, interactive tutorial authoring can include tracking of hand position, and project green and red onto end users' hands to indicate correct and incorrect positioning respectively. DuploTrack allows users to author a tutorial for creating Duplo block models using depth sensing to infer positions and rotations automatically, and projective guidance is available to end users of the tutorial. Our design of the AuthAR system was grounded by our exploration of the assembly task design space, and a study of related research and investigation into the difficulties associated with the process of generating tutorials (both for "traditional" media, as well as AR). Below we describe the design guidelines we followed when making decisions about the implementation of our system. It is important that the author and assembler be able to perform the assembly task without being burdened by the tutorial apparatus or interface. Though many AR/VR interfaces are either mediated by a handheld device or require use of freehand gestures for input, assembly tasks often require the use of both hands in parallel. For this reason, we prioritize hands-free interaction with the system such that users can always keep their hands free for assembly. Prior studies have shown that different representations (text, static pictures, video, animations, etc.) can all be valuable for people following along with a tutorial. Our system should allow authors to document their tutorial using multiple media types to best capture the necessary information. Manual content creation in AR allows for high expressivity but is time-consuming and can add complexity. Automatic content creation tools are easier to use, however have the side effect of limiting the author's creative control. Our tool should let authors move been automatic and manual creation modes to get the benefits of both models. In the most "automatic" case, an author should be able to generate a tutorial by doing little more than simply completing the assembly task as they would normally. With the ability to generate, refine and augment tutorial step representations, our tool should allow authors to create the tutorial while performing the task and make necessary tweaks directly after each step. This form of contextual and in situ editing allows authors to add callouts or make changes while they are easy to remember and reduces the need for post-processing of tutorials at a desktop computer. The design space of tutorials for assembly tasks in augmented reality is broad, with many dimensions of variability both in how the instructions are presented to an end-user and how they are authored. To inform the design of our system and explore how to best achieve our design goals, we first mapped out some areas of this design space most relevant to our work (Figure 2). Perhaps the most important dimension of variability in the AR-based assembly task tutorial design space is how the tutorial information is presented within the AR environment. Like more "traditional" tutorial delivery mediums, the AR environment is able to present static text and images, as well as show explanatory videos. A most straightforward (and still useful) application of AR technology for sharing assembly tutorials would be to display relevant information about the task as a heads-up display (HUD) in the user's headset, leaving their hand's free to perform the task. Unique to AR however is the ability to spatially associate these "traditional" elements with points or objects in the physical space. Further, an AR tutorial can present the assembly instructions "live", by displaying assembly guidance graphics or animations interleaved into the physical space. With our system we chose to use traditional text, pictures, and video presentation methods in addition to dynamic instructions, since they each have their unique benefits (D2). In order for the HoloLens to listen for the keyword "Stop Recording" during first/third person video recording, it cannot simultaneously record dictation to complement the video. For this reason, the current form of AuthAR records muted videos, but future iterations with additional microphone input capability would rectify this. With use of audio in content authoring infeasible, text serves to supplement muted video. When it comes to authoring the tutorial, the content could be constructed in situ, that is, in the same physical space as the task is being performed, or in an external secondary location such as a desktop computer. Our work explores in situ authoring to reduce the required effort (D3) and maintain context (D4). The content for the tutorial can be automatically captured as the authors moves through the assembly steps, or manually created by explicitly specifying what should be happening at each step. Automatically generating the tutorials streamlines the authoring and content creation process, but limits expressivity of the tutorial instructions. We automatically capture as much as possible, while allowing manual additions where the author thinks they would be helpful (D3). Another aspect of tutorial authoring is how the content is edited. A traditional practice is to capture the necessary content first, then go back and edit for time and clarity in post-processing. Alternately, the content can be edited concurrently with the process of content collection-which if implemented poorly could negatively impact the "flow" of the creation process. In the best case, this in having a completed tutorial ready to share as soon as the author has completed the task themselves. We focus on creating a welldesigned concurrent editing process (D1). Interacting with the tutorial system, both in the creation of the tutorial as well as when following along, can be accomplished through many means. AR applications can use touch controls, gestures, or dedicated hardware controllers. However, for assembly tasks it is desirable to have both hands available at all times for the assembly task -rather than have them occupied by control of the tutorial authoring system (D1). For that reason, we exclusively use voice and gaze controls. AuthAR is a suite of software tools that allow tutorial authors to generate AR content (Figure 1). Optitrack motion capture cameras visually track marked assembly materials and a screwdriver, adding changes to position, transform and locations of screws added with a tracked screwdriver to the tutorial automatically. The HoloLens captures first person videos and images and a mounted Android tablet simultaneously captures third person video. The HoloLens also guides authors through the process of creating the tutorial and allows for gaze-and voice-based interaction to add and update content. This includes addition of augmentations such as location-specific callout points, marking locations of untrackable screws, and specification of images as "negative examples" (steps that should be avoided) or warnings. AuthAR's system architecture consists of three components-the Microsoft HoloLens, a Samsung Tab A 10.1 Android tablet and a server running on a desktop computer all on the same network (Figure 3). As a proxy to sufficient object recognition and point-cloud generation directly from the headset, we use relatively small (~1cm) Optitrack visual markers for detection of material position and rotation. In developing AuthAR, we envisioned headsets of the future having onboard object recognition, though as a proxy to such capabilities, we implement a networked system to coordinate object positions generated by Optitrack's Motive software and the HoloLens to make the physical objects interactive. To provide this interactivity, the HoloLens generates virtual replicas of each piece and overlays these models as invisible renderings at the position and rotation of the tracked object. This invisible object takes the same shape as the physical object but has its visual rendering component disabled. It engages the HoloLens raycast and gives the user the illusion of placing virtual augmentations on physical objects. Additionally, we add a tracked handle to a screwdriver for tracking to infer screw events. The server connects to the Optitrack host via UDP and continues to update object and tool transforms. This server sends the data to the HoloLens via 64-character messages and the HoloLens' representation of the transform is updated accordingly. The Android tablet simply serves to record 3 rd person video of the tutorial. When the HoloLens starts and stops recording 1 st person demonstration, a message is passed to the server and then to the Android tablet to toggle recording. Throughout the paper, we demonstrate usage of AuthAR to build a tutorial for assembly of an Ikea laptop stand. Parts have been outfitted with Optitrack visual markers and defined as rigid bodies in Optitrack's Motive software. Simple representations of these parts are passed to the HoloLens as invisible colliders, so the physical components act as interactive objects in AR (Figure 4). Though this configuration is specific to the laptop stand, this approach is easily extensible to other assembly workflows. Users simply define combinations of visual markers as rigid bodies and build simplified models of the individual parts. In this case, these models are combinations of cubes that have been scaled along X, Y and Z axes, giving rough approximations of the parts' shapes. This initial step of predefining the shapes of these parts allows fully in situ editing allows for the remainder of the process. To avoid encumbering the assembly process with tutorial generation steps, interaction with AuthAR involves only voice, gaze, and use of materials and tools. This allows the user to always keep his/her hands free to build. By using only voice-and gaze-based interaction, we also eliminate the need for an occluding visual interface of menus and buttons, avoiding interference with the physical tutorial tasks. For flexibility, the user can add augmentations at any point while building the tutorial. However, to encourage faster onboarding, we guide the user through a two-phase process for each step: Step Recording and Step Review. When the user says "Start Recording" AuthAR begins recording changes to object transforms such that moving the physical objects maps directly to manipulating the virtual representations of those objects in the tutorial. This command also initiates video recording using the HoloLens' built-in camera and the tablet's 3 rd person perspective (Figure 5). AuthAR also records when the screwdriver's tip comes in contact with an object piece and generates a screw hole on that object (though not displayed until the Review Mode). To do this we add a tracked attachment to the handle, similar to what was done for Dodecapen and SymbiosisSketch to provide the position of the tip based on the orientation of the attachment. Given real-time streaming of position data of the screwdriver's handle and a priori knowledge of the length from the handle to the tip as 10.5 cm, we calculate the position of the screwdriver's tip as 10.5 cm forward from the handle. When the user says "Finish Recording", the HoloLens prompts the user for a step description and records dictation. When description is complete, the user enters an idle state until they are ready for review. Step Review: Manually Added Features After a tutorial author has completed a step recording, they can they can say "Review Step" to enter review mode for the step. The 1 st person video just recorded is played on a loop across from the author, automatically repositioning itself such that the author can look up at any point and see the video directly in front of him/her. This allows the author to draw upon their own experience when adding manual augmentations. Existing augmentations shift into focus by getting larger or expanding when the user is looking closer to that augmentation than any other-this eliminates the need to focus on small points (~3cm) to engage with them. When looking toward a particular object, the available commands to update the augmentation are shown in the top right of the Heads-up Display. After recording the step, the tutorial author may want to draw the tutorial user's attention to a particular point, similar to how this is done on 2D paper-based instructions (Figure 6). To do this, the author focuses the gaze-based cursor on the specific point on the tracked object where the callout point should go and says "Add Point". This adds a small virtual sphere anchored by its relative position on the object. The process of adding a captioned image (Figure 7), begins when the user looks toward a callout point and says "Add Picture". The system then starts a countdown from three in the middle of the heads-up display to indicate that a picture is about to be taken, and then displays "Hold still!" when the countdown is complete, at which point the HoloLens captures the current frame. The collected image is saved and immediately loaded over the callout point. The author can then say "Add Text", and a prompt in the middle of the heads-up display shows "Speak the image caption" and the HoloLens begins listening for recorded dictation. After a pause of 3 seconds, the HoloLens finishes recording and immediately associates the spoken text as the image caption. Authors can describe a callout point as a negative example or a warning simply by looking toward the callout point and saying "Warning". This defines that callout point as a warning or negative example and will draw tutorial users to be extra attentive when traversing the tutorial. The callout point turns red and the associated image is given a red border around it (Figure 8). Authors are also able to move the point along the surface of any tracked piece or delete it entirely. Though fastening components together is an important component of an assembly tutorial, the objects are too small to track with traditional object tracking technologies. During step recording, AuthAR records use of the tracked screwdriver and detects when it was used on a tracked material, generating a screw hole. Making use of these generated screw holes from step recording, the author can associate a virtual screw with the hole by looking toward the hole and saying "Add Screw" (Figure 9). The user cycles through possible screws by saying "Next" and "Previous"-commands which cycle through possible virtual screw representations. The author is able to hold the physical screw up to the virtual one for comparison and say "This One" to associate the screw with that hole. Authors can also manually add new fasteners in areas that cannot be tracked. To do so, the author once again says "Add Screw", pulling up the same menu, but when a screw has been selected, the ray-casted, gaze-based cursor allows the user to manually place the virtual screw where it was physically placed. This is useful for the laptop stand, for example, as there are rubber feet that need to be screwed in by hand, rather than with the tracked screwdriver. Saying "Finish Review" brings the user back to the original idle state and the user has iterated through the step building process at this point. Toward validating AuthAR, we discuss our initial observations in testing with tutorial authors, present an example application that parses and displays the generated tutorial for end users, and explain extensibility beyond the presented use case. In doing so, we consider improvements to AuthAR, and design considerations for other in situ AR content authoring tools. To gather initial observations and feedback of the system, we asked two users to generate a tutorial for an Ikea laptop stand. Though we have not formally evaluated AuthAR for usability, this initial feedback provides some insight into possible improvements to AuthAR and generalized takeaways in building such systems. The standard paper instructions for the stand consists of four steps: fastening the legs to the bottom base, fastening the top to the legs, adding screw-on feet to the bottom of the structure and adding glass to the top. We guided the users through using the system while they created an AR tutorial for assembling this piece of furniture. Both testers found the system helpful for generating tutorials, one noting that "being able to take pictures with the head for annotations was really useful." This suggests that the embodied gaze-based interaction is particularly well-suited to picture and video recording. Most of the functionality for making refinements in the tutorial is enabled by the user looking anywhere near the objects, however, adding new callout points requires accurate hovering of the cursor on the object of interest while speaking a command. One user mentioned that it was "kind awkward to point at certain points with the head". In such systems that require precise placement of virtual objects on physical components, pointing at and touching the position where the callout point should go would be a useful improvement over a gaze-only approach. Though fulfilling the hands-free requirement of a tutorial generation system (D1), AuthAR's use of dictation recognition for text entry was particularly challenging for users, in part due to the automated prompting for step descriptions and titles. One participant was surprised by the immediate prompt for a step description, and that "it was hard to formulate something articulate to say by the time it had finished recording", so future iterations will likely incorporate users explicitly starting dictation recognition for a title so they are prepared to give one. While we focus on authoring the tutorial in real time, the users wanted to view and refine previous steps. One possible enhancement to an in situ content authoring tool would be a "navigation prompt to let you jump between steps and review your work." With AuthAR, this would provide users the ability to review and refine the full tutorial at a high level, including previous steps. Figure 9. User adding virtual screws to the tutorial. The user can hold the physical screw up to the virtual one for comparison (top), and if the screw hole was not automatically generated, the user can place the screw via ray-cast from the headset (middle). Though we focus on authoring of assembly tutorials, we also implemented and tested a playback mode to validate AuthAR's ability to generate working tutorials. The tutorial author can save a tutorial after finishing a step, and AuthAR generates an XML representation of the entire tutorial. We load this XML into the playback application, also built for the HoloLens, and components of the steps built earlier are loaded and displayed for the end user to follow. The guidance techniques are simple but demonstrate success of the authoring process and portability of the generated tutorial. This application projects lines from each piece's location to where that piece needs to go (Figure 10a). When the end user of the tutorial has correctly placed the object, a notification appears in front of them (Figure 10b). The user also receives guidance of where add screws and when the playback application recognizes a screw event in that location, the screw augmentation disappears (Figure 10c). Both first person and third person video representations play on a loop across the table from the user (Figure 10d). As the user walks around the table, the videos adjust their positions such that the user can always look up and see the videos straight across from them. Use of the third person video is currently the only tutorial component that requires postprocessing after the in situ authoring process is complete. Because the XML representation of the tutorial uses file paths for videos, the author needs to manually move the video from the Android tablet to the headset's file system. Future iterations could automatically stream these videos between components. We demonstrate AuthAR with assembly of an Ikea laptop stand but note that it could be extended to any physical task where simplified virtual models of the pieces could be built or obtained. All of the pieces used for the laptop stand were scaled cubes or combinations of scaled cubes, with disabled "Renderer" components to create the illusion of adding virtual objects to physical parts, when in reality, there were invisible objects overlaid on top of the physical pieces. So simply loading in pieces and turning any visual rendering allow for extensibility to virtually any assembly task. While demonstrated with an augmented screwdriver, AuthAR could be extended to support different tools, or an integrated workspace of smart tools. We employed very simple logic, that whenever the tip of the screwdriver hovers near a piece, AuthAR adds a screw hole. Future iterations could employ more advanced logic for detecting tool usage. The need for a large truss with 10 Optitrack cameras enables very accurate tracking of visual markers but limits the AuthAR's widespread deployment in its current state. For practical use we envision the localization of objects being done with the headset only, or with cheaper external optical tracking, perhaps with black and white fiduciary markers. In this scenario, tracked pieces could be established in real time though user-assisted object recognition, rather than defining their shapes prior to running AuthAR. The in situ authoring approach offered by AuthAR allows the user to concurrently craft the tutorial while assembling the pieces. However, the gaze/voice multimodal interface does not provide users with efficient tools to fine-tune the generated tutorial. To this end, AuthAR should be complimented by a 2D interface for precise editing of individual components of the tutorial, similar to tutorial generation tools described in previous work. Trimming 1 st and 3 rd person perspective videos, cropping images and editing text are currently not well-supported by AuthAR and would be better suited to a mouse and keyboard interface. This complimentary approach would also allow users to focus on coarse-grain tutorial generation and object assembly without being burdened by these smaller edits that can easily be done after the fact. AuthAR enables tutorial authors to generate mixed media tutorials semi-automatically to guide end users through the assembly process. We automatically record expert demonstration where possible and allow for in situ editing for refinements and additions. We built AuthAR with several design guidelines in mind, validated with the authoring of a tutorial for assembling a laptop stand, and discuss the extensibility to assembly of other tasks by simply loading different virtual models into AuthAR. We see AuthAR enabling authoring of tutorials that could reach a widespread population with mixed media tutorials flexible to the preferences of each individual user.

We present a mixed media assembly tutorial authoring system that streamlines creation of videos, images, text and dynamic instructions in situ.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We automatically extract fingering information from videos of piano performances, to be used in automatic fingering prediction models.