Question: Represent the scientific article to find a short TLDR

Question: Many large text collections exhibit graph structures, either inherent to the content itself or encoded in the metadata of the individual documents. Example graphs extracted from document collections are co-author networks, citation networks, or named-entity-cooccurrence networks. Furthermore, social networks can be extracted from email corpora, tweets, or social media. When it comes to visualising these large corpora, either the textual content or the network graph are used. In this paper, we propose to incorporate both, text and graph, to not only visualise the semantic information encoded in the documents' content but also the relationships expressed by the inherent network structure. To this end, we introduce a novel algorithm based on multi-objective optimisation to jointly position embedded documents and graph nodes in a two-dimensional landscape. We illustrate the effectiveness of our approach with real-world datasets and show that we can capture the semantics of large document collections better than other visualisations based on either the content or the network information. Substantial amounts of data is produced in our modern information society each day. A large portion of it comes from the communication on social media platforms, within chat applications, or via emails. This data exhibits dualtiy in the sense that they can be represented as text and graph. The metadata provides an inherent graph structure given by the social network between correspondents and the exchanged messages constitute the textual content. In addition, there are many other datasets that exhibit these two facets. Some of them are found in bibliometrics, for example in collections of research publications as co-author and citation networks. When it comes to analyse these types of datasets, usually either the content or the graph structure is neglected. In data exploration scenarios the goal of getting an overview of the datasets at hand is insurmountable with current tools. The sheer amount of data prohibits simple visualisations of networks or meaningful keyword-driven summaries of the textual content. Data-driven journalism often has to deal with leaked, unstructured, very heterogeneous data, e.g. in the context of the Panama Papers, where journalists needed to untangle and order huge amounts of information, search entities, and visualise found patterns . Similar datasets are of interest in the context of computational forensics . Auditing firms and law enforcement need to sift through huge amounts of data to gather evidence of criminal activity, often involving communication networks and documents . Users investigating such data want to be able to quickly gain an overview of its entirety, since the large amount of heterogeneous data renders experts' investigations by hand infeasible. Computer-aided exploration tools can support their work to identify irregularities, inappropriate content, or suspicious patterns. Current tools 1 lack sufficient semantic support, for example by incorporating document embeddings and the ability to combine text and network information intuitively. We propose MODiR, a scalable multi-objective dimensionality reduction algorithm, and show how it can be used to generate an overview of entire text datasets with inherent network information in a single interactive visualisation. Special graph databases enable the efficient storage of large relationship networks and provide interfaces to query or analyse the data. However, without prior knowledge, it is practically impossible to gain an overview or quick insights into global network structures. Although traditional node-link visualisations of a graph can provide this overview, all semantic information from associated textual content is lost completely. Technically, our goal is to combine network layouts with dimensionality reduction of highdimensional semantic embedding spaces. Giving an overview over latent structures and topics in one visualisation may significantly improve the exploration of a corpus by users unfamiliar with the domain and terminology. This means, we have to integrate multiple aspects of the data, especially graph and text, into a single visualisation. The challenge is to provide an intuitive, two-dimensional representation of both the graph and the text, while balancing potentially contradicting objectives of these representations. In contrast to existing dimensionality reduction methods, such as tSNE , MODiR uses a novel approach to transform high-dimensional data into two dimensions while optimising multiple constraints simultaneously to ensure an optimal layout of semantic information extracted from text and the associated network. To minimise the computational complexity that would come from a naive combination of network drawing and dimensionality reduction algorithms, we formally use the notion of a hypergraph. In this way, we are able to move repeated expensive computations from the iterative document-centred optimisation to a preprocessing step that constructs the hypergraph. We use real-world datasets from different domains to demonstrate the effectiveness and flexibility of our approach. MODiR-generated representations are compared to a series of baselines and state-of-the-art dimensionality reduction methods. We further show that our integrated view of these datasets exhibiting duality is superior to approaches focusing on text-only or network-only information when computing the visualisation. With MODiR we bridge the gap between text and network visualisation by jointly reducing the dimensionality of the input data. Therefore we subdivided this part into three sections to highlight related work in the areas of text visualisation, representation learning, as well as dimensionality reduction. Other work that tries to jointly model text and networks but without dimensionality reduction and without a focus on visualisation is LINE. They generate information networks consisting of different types of nodes, e.g. words from document content and authors from document metadata. Another tool that investigates combining graph structure with textual elements is VOSviewer . They construct and visualise bibliographic networks that provide a multi-view interface to explore and filter keywords and network aspects of such datasets. In our work we go beyond building a network from textual data but instead project the textual data into a latent space. Document visualisation aims to visualise the textual content, such that users gain quick insights into topics, latent phrases, or trends. Tiara extracts topics and derives time-sensitive keywords to depict evolving subjects over time as stacked plots. Another line of work projects documents into a latent space, for example by using topic models or embeddings: Creating scatterplots of embedded documents of a large corpus may in a very dense and unclear layout, so Chen et al. developed an algorithm to reduce over-full visualisations by picking representative documents. A different approach is taken by Fortuna et al. , who do not show documents directly, but generate a heatmap of the populated canvas and overlay it with salient phrases at more densely populated areas from the underlying documents in that region. Friedl et al. extend that concept by drawing clear lines between regions and colouring them. They also add edges between salient phrases based on co-occurrences in the texts. A map analogy can be used to visualise the contents of documents by embedding them into a high dimensional semantic space and projecting it on a two-dimensional canvas as a document landscape. Most recently Cartograph was proposed, which is visually very similar to previous approaches, but pre-renders information at different resolution and uses a tiling server with (geographic) map technology to deliver responsive interactions with the document landscape. Regions are coloured based on underlying ontologies from a knowledge-base. Networks are traditionally visualised using so-called node-link graphs. This way, any additional information related to nodes and edges are lost. The layout of nodes usually follows a force-based analogy first proposed by. Newer approaches optimise the compu-tational complexity and include local metrics to better represent inherent structures as for example ForceAtlas2 , which is the default network algorithm for the network visualisation tool Gephi. The text and network visualisation methods discussed above primarily use structural properties of the data to generate their layout. Although we focus on the visualisation of text data with inherent graph information, MODiR can work with arbitrary kinds of data. Our model only requires a way to project the data into a high-dimensional Euclidean vector space so that the distance between two points can be interpreted as their (semantic) similarity. Traditionally, text can be represented as bagof-words vector that optionally is weighted by respective tf-idf scores. In recent years, embeddings became more popular as they conserve semantic meaning in their vector representation. introduced neural architectures to learn high-dimensional vector representations for words and paragraphs . Similar methods are used to learn representations for nodes in a network based on either the structural neighbourhood or additional heterogeneous information . Schlötterer et al. attempted to learn joint representations of network structure and document contents but saw no improvement over conventional models in a series of classification tasks. We only use the structural information of the network for better control over fine-grained adjustments in our layout algorithm. The goal of dimensionality reduction is to represent high-dimensional data in a low-dimensional space while preserving the characteristics of the original data as sound as possible. A very common application of dimensionality reduction is to project high-dimensional data into two dimensions for the purpose of visual interpretation. Generally, these methods follow one of three mathematical models. Linear models, such as Principle Component Analysis (PCA) can be calculated very efficiently and have proven to reduce input spaces to improve the performance of downstream tasks. Thus, they are often indirectly used for feature extraction. Although reductions to two dimensions for visualisations are appropriate for quick initial data exploration, other approaches are able to better preserve data characteristics in two dimensions. For example, the nonlinear Sammon mapping tries to preserve the structure of inter-point distances in high-dimensional space in low-dimensional space. The ing visualisations are generally better then PCA to show relatedness of individual data points. Lastly, there are probabilistic models like Stochastic Neighbour Embeddings (SNE) . They are similar to a Sammon mapping in that they use inter-point distances but model these distances as probability distributions. The t-distributed SNE has proven to produce competitive for visualising datasets while preserving characteristics , however its nondeterministic nature may produce greatly varying . Recently, FltSNE was proposed, an optimisation of tSNE that significantly reduces the computational complexity. Other newer dimensionality reduction algorithms like LargeVis and UMAP scale almost linearly by using efficient nearest neighbourhood approximations in the high-dimensional space and spectral embeddings to initialise positions of points in the low-dimensional space to reduce the number of fine-tuning iterations. Visualisations of complex datasets are restricted to two or three dimensions for users to grasp the structure and patterns of the data. We integrate multiple entities (i.e., documents and persons) into a joint visualisation, which we call landscape. This landscape consists of a base-layer containing all documents depicted as dots forming the document landscape; nodes and their connections are placed on top of this base-layer as circles connected by lines forming the graph layer. In this section, we propose the MODiR algorithm which integrates multiple objectives during the layout process to find an overall good fit of the data within the different layers. Our approach is derived from state-ofthe-art methods for drawing either the network layer or the document landscape. We assume that documents are given as high-dimensional vectors and entities are linked among one another and to the documents. These links are used as restrictions during the multi-objective dimensionality reduction of document vectors. Let x (i) ∈ X ⊂ R d be the set of n documents in their d-dimensional representation and y (i) ∈ Y ⊂ R 2 the respective positions on the document landscape. Let H(V, E) be a hypergraph based on the network information inferred from the document corpus, with vertices V = X P, where X are the documents and p i ∈ P are the entities in the network and hyperedges e k ∈ E describing the relation between documents and entities. For each pair of entities p m, p n ∈ P that are connected in the context of documents x (i),... ∈ X, there is a hyperedge e k = {p m, p n, x (i),...}. Analogously, the same definition applies to Y. Further, H Y or H X is used to explicitly state the respective document representation used. The position in the graph layer π: P → R 2 of an entity p m is defined as where E pm ⊂ H Y is the set of hyperedges containing p m and N pm is the number of documents p m is associated with. 2 This effectively places an entity at the centre of its respective documents. More elaborate methods like a density-based weighted average are also applicable to mitigate the influence of outliers. For simplicity we will abbreviate π(, where W ∈ R 2×n is the projection matrix leant by MODiR based on multiple objectives ϕ {1,2,3} using gradient descend, as defined later in this section. The objectives are weighted by manually set parameters θ {1,2,3} to balance the effects that favour principles focused on either the graph layer or the document landscape, as they may contradict one another. Given a high-dimensional hypergraph H X, the matrix W, and a entity projection π, we define the ing multi-objective dimensionality reduction function as In the following paragraphs, we will formally introduce MODiR's objectives. Objectives and are inspired by tSNE and use the neighbourhood context of documents in X to position similar documents near one another and unrelated ones further apart in Y. Objective attracts documents based on co-occurrence in hyperedges so that the ing π m will be closer if they are well connected in the graph. This third objective also implicitly brings documents closer to their respective entities. Objective: Similar documents are near one another. Semantically similar documents should be closer on the document landscape and dissimilar ones further apart. To measure the semantic similarity of documents, used a naïve bag-of-words representation. Although tSNE preserves the inherent semantic structure in two-dimensional representations from these sparse vectors , we opted to use document embeddings. This has the advantage that, when only part of the data is visualised, the embedding model can still be trained on a larger set of documents and thus retain the additional information. Objective is inspired by the efficient usage of context words in word2vec . Corresponding to the skip-gram model, we define the context X k,x (i) ⊂ X of a document x (i) by its k nearest neighbours in the embedding space. The first objective is defined as with σ being the sigmoid function. Distances are normalised based on the context to make them comparable between the high-dimensional and two-dimensional space and rescaled by the sigmoid. Objective: Dissimilar documents are apart from one another. The optimal solution to the previously defined objective would be to project all documents onto the same point on the twodimensional canvas. In order to counteract that, we introduce negative examples for each pair of context documents. We do so by sampling a set of l documents that are not in the k neighbourhood of be the set of negative samples for x (i), then the second objective is defined as 2 Np m:= {x Objective: Connected entities are near one another and their documents. This object serves two purposes: All documents y (i) associated with a person p m are placed near its π m position in the graph layer and two people π m and π n are forced near one another if they are connected. Let E y (i) ⊂ E be the set of hyperedges in the hypergraph H containing the document y (i) and E Y y (i) = e k ∈E y (i) e k \ P all documents that are linked to y (i) through an entity, then the third objective is defined as which, when minimised, attracts documents that are related through entities. This has two implicit effects: An entity p m gets closer to its documents as they are attracted to π m without having to explicitly compute this position using Equation 1. Also, related entities p m, p n are attracted to one another since they appear in the same hyperedges. The computational complexity of this objective is strongly related to the connectedness of entities in the graph. For dense graphs, we propose a heuristic by only using a subset of s documents from the context E Y y (i) of y (i). An objective modelling a repulsive force as in force-directed graph layouts is not needed as the first two objectives ϕ {1,2} provide enough counteracting force. Algorithm. The positions of entities and documents on the landscape are calculated using the previously defined objectives as follows. First, we construct the hypergraph H X with document contexts including the set of k-neighbourhoods X k,x (i). Relevant pairwise distances can be stored in an adjacency matrix so reduce computational overhead in Equations 2 and 3. For more efficient training, the randomly sampled l negative neighbourhoodsX can be prepared ahead of time and then only masked during later. The s-neighbourhoods for entities in Equation 4 E Y y (i) can only be prepared with references, as Y y (i) updates with each iteration. We designed the algorithm to move as much repetitive computations to pre-processing ahead of time or each epoch. Creating these sets is very efficient using Hierarchical Navigable Small World graphs (HNSW) for approximate nearest neighbour search. Overall we are able to reduce the pre-processing complexity to O(n log n) and for each iteration O(kln), with k, l n near linear. After generating the context sets, we use gradient descend to update the projection matrix W with learning rate η reducing the overall error Φ as defined by Selecting appropriate values for the hyperparameters k, l, s, and θ {1,2,3} is critical to produce meaningful . We found l = k in all experiments to produce the best as this way for every similar document the model has one dissimilar document to compare. Inspired by tSNE , we limit hyperparameters by setting k and s dynamically for each document based on a user-defined perplexity. With these adaptations, the only parameters to be set are the perplexity β that roughly determines the context size, the learning rate η, and the objective weights, which can often stay at a default setting. A reference implementation including a modular processing pipeline for different datasets, approaches, and experiments is available on GitHub 3. Our approach is mainly motivated to explore business communication data (namely emails), such as the Enron corpus . However, due to the lack of ground truth, we will focus our evaluation on research publications and their co-authorship network. Results of dimensionality reduction can be subjective, so as in prior work on dimensionality reduction (; ;), we will qualitatively compare our approach to a variety of baselines and provide some quantitative experiments. Experimental Setup. Here, we are using the Semantic Scholar 4 Open Corpus (S2) , we remove articles with missing information and limit to the six communities Data Mining, Databases, Machine Learning, NLP, Computer Vision, and HCI. This way we discard clearly unrelated computer science articles and biomedical studies for a more fine grained analysis. For in-depth comparisons we use an even smaller subset from S2 of 24 hand-picked authors, their co-authors, and their papers (S2b). To our knowledge, there are no algorithms that use multiple objectives for dimensionality reduction of high-dimensional data. Popular approaches for traditional dimensionality reduction are tSNE and PCA. As baselines, we use the original optimised implementation of tSNE 7 written in C as provided by the authors. The quantitative evaluation is two-fold: MODiR can simulate their behaviour by setting θ 3 = 0 and thus ignoring the objective that incorporates network information. In our experiments we use the following parameter settings. For tSNE we set the perplexity to P erp(P i) = 5, θ = 0.5 and run it for 1,000 iterations. In MODiR we set the neighbourhood size to k = s = 10, the negative context size to l = 20, and all objective weights θ {1,2,3} = 1.0. For a discussion on the influence of hyperparameters, we refer to the supplemental material. The speed of convergence depends on the learning rate η and thus dictates the number of maximum iterations. Early stopping with a threshold on the update rate could be implemented. Depending on the size of the dataset and a fixed learning rate of η = 0.01, MODiR generally converges after 10 to 200 iterations, for larger and more connected data it is advisable to use a higher learning rate in the first epoch for initialisation and then reducing it to very small updates. For better comparability, we use a constant number of iterations of T = 100. Quantitative Evaluation. state, it is by definition impossible to fully represent the structure of intrinsically high-dimensional data, such as a set of document embeddings, in two dimensions. However, stochastic neighbour embeddings are able to capture intrinsic structures well in two dimensional representations . To measure this capability, we compare the ability of k-means++ to cluster the high-and twodimensional space. We set the number of clusters to the number or research communities (k = 6) and calculate the percentage of of papers for each community per cluster. Therefore we assign each community to the cluster with most respective papers and make sure to use a clustering with an even distribution. Results are listed in Table 1 for tSNE, PCA, MODiR, and the original high dimensional embedding averaged over five runs. We see, that as expected due to topical overlap of communities, even original embeddings can't be accurately clustered. Interestingly though, there seems to be a significant difference between AM and S2 although the sets of papers intersect, which we assume is due to the fact, that S2 is larger and additionally contains more recent papers. Although PCA (S2b), subsampled for readability; (a) the network is laid out first, documents are randomly placed along edges; (b) the network is laid out first, documents are randomly placed around nodes; (c) documents are part of the network layout as nodes in the graph that replace author-author edges; (d) the document landscape is laid out first, nodes are positioned at the centre of their associated documents; (e) tSNE is applied on papers and authors together, where documents are aggregated to represent authors often does not generate visualisations in which classes can be clearly distinguished, the clustering algorithm is still able to separate them with competitive compared to tSNE and MODiR. MODiR not only aims to produce a good document landscape, but also a good layout of the network layer. Graph layouts are well studied, thus we refer to related work on aesthetics and readability . While these are very elaborate and consider many aspects, we decided to use Noack's normalised AtEdge-length : It describes how well the space utilisation is by measuring whether edges are as short as possible with respect to the size and density of the graph. Table 1 contains the . Although the AtEdge metric is comparable for layouts of the same graph, it is not comparable between datasets as can be seen by the fact, that a larger number of edges causes an overall lower score. The AtEdge length produced by PCA is generally better than that of tSNE while MODiR outperforms both as our approach specifically includes an optimised network layout. The better performance of PCA over tSNE can be explained by the ing layouts being more densely clustered in one spot. Although the AtEdge length aims to give a lower score for too close positioning, it is not able to balance that to the many very long edges in the layout produced by tSNE. Qualitative Evaluation. Apart from a purely quantitative evaluation, we use the hand-selected Semantic Scholar dataset (S2b) to visually compare compare network-centric baselines (a-c), document-focused baselines (d-e) and MODiR (f) in Figure 1. Papers are depicted as circles where the stroke colour corresponds to the communities, black lines and dots are authors and their coauthorships, size corresponds to the number of publications. For better readability and comparability, the number of drawn points is reduced and three communities are marked. In Figure 1a we use the weighted co-authorship network drawn using and scatter the papers along their respective edges after the graph is laid out. We see, that active collaboration is easy to identify as densely populated edges and research communities of selected areas are mostly coherent and unconnected researchers are spatially separated from others. Although it is possible to distinguish the different communities in the graph layer,the document landscape isn't as clear. The ML researchers are split apart from the rest of the NLP community, which in turn is overcrowded. Figure 1b uses the same network layout but places articles randomly around their first author, which makes it easy to spot the scientific communities by colour. Lastly, we include papers as nodes and co-authorship edges are connected through them during the network layout in Figure 1c. This produces a very clean looking layout compared with the other baselines, however papers lump together and are not evenly distributed. Furthermore, semantic nuances between papers are mostly lost which becomes most apparent in the now separated database clusters. Also, the semantic overlap between the ML and NLP communities is not noticeable. Figure 1d positions documents using tSNE and places researchers using Equation 1. We see that articles are positioned on the landscape so that research areas are distinctly recognisable by colour. Papers that could not be assigned to a specific area are scattered across the entire landscape. The collaboration network is laid out surprisingly good. The research interests of the authors are coherent between the network and the document landscape, it even shows the close relation between NLP and ML, while showing a clear separation to database related topics. Nonetheless, the network should be loosened for better readability, for example members of the same research group who frequently co-author papers tend to collide. Unconnected authors are almost not visible as they drift toward densely populated areas in the middle. In Figure 1e, we included authors as virtual documents as the sum of their papers during the tSNE reduction. This shows some improvement, as the network layout is more loose and fewer edges overlap and the issue with collapsing research groups is also mostly mitigated. The semantic overlap of ML and NLP is niceley captured along with the difference to the database papers. However, the network is not clearly readable. With MODiR the three research communities become clearly distinguishable, both in the graph layer and in the document landscape. Nodes of well connected communities are close together, yet are not too close locally, and separate spatially from other communities. The document landscape is laid out more clearly, as papers from different fields are grouped to mostly distinct clusters. Obviously there is still a slight overlap as a of semantic similarities. As previously pointed out, this visualisation also correctly reveals, that the ML and NLP communities are more closely related to each other (both use machine learning) than to DB. The authorship of documents however can only be conveyed through interaction, so this information is not present in the static visualisations shown here. Based on these we argue, that the network information improves the (visual) community detection. The document embeddings of articles can only reflect the semantic similarities, which may overlap. In conjunction with information from the co-authorship network, the embeddings are put into their context and thus are more meaningful in a joint visualisation. In this paper we discussed how to jointly visualise text and network data with all its aspects on a single canvas. Therefore we identified three principles that should be balanced by a visualisation algorithm. From those we derived formal objectives that are used by a gradient descend algorithm. We have shown how to use that to generate landscapes which consist of a base-layer, where the embedded unstructured texts are positioned such that their closeness in the document landscape reflects semantic similarity. Secondly, the landscape consists of a graph layer onto which the inherent network is drawn such that well connected nodes are close to one another. Lastly, both aspects can be balanced so that nodes are close to the documents they are associated with while preserving the graph-induced neighbourhood. We proposed MODiR, a novel multi-objective dimensionality reduction algorithm which iteratively optimises the document and network layout to generate insightful visualisations using the objectives mentioned above. In comparison with baseline approaches, this multi-objective approach provided best balanced overall as measured by various metrics. In particular, we have shown that MODiR outperforms state-of-the-art algorithms, such as tSNE. We also implemented an initial prototype for an intuitive and interactive exploration of multiple datasets. with over 45 million articles. Both corpora cover a range of different scientific fields. Semantic Scholar for example integrates multiple data sources like DBLP and PubMed and mostly covers computer science, neuroscience, and biomedical research. Unlike DBLP however, S2 and AM not only contain bibliographic metadata, such as authors, date, venue, citations, but also abstracts to most articles, that we use to train document embeddings using the Doc2Vec model in Gensim 10. Similar to Carvallari et al. remove articles with missing information and limit to six communities that are aggregated by venues as listed in Table 3. This way we reduce the size and also remove clearly unrelated computer science articles and biomedical studies. For in depth comparisons we reduce the S2 dataset to 24 hand-picked authors, their co-authors, and their papers (S2b). Note, that the characteristics of the networks differ greatly as the ratio between documents, nodes, and edges in Table 2 shows. In an email corpus, a larger number of documents is attributed to fewer nodes and the distribution has a high variance (some people write few emails, some a lot). In the academic corpora on the other hand, the number of documents per author is relatively low and similar throughout. Especially different is the news corpus, that contains one entity that is linked to all other entities and to all documents. In Section 4 we gave brief overview of the settings used in the experiments presented above. Here, we provide additional insights from our experiments on different hyperparameter settings for MODiR. The context sizes are the most important parameters. The first two objective weights can be ignored, as the context size has similar effects, so we set θ 1 = θ 2 = 1.0 in all our experiments. Generally, small numbers for k, l, s perform better. This is in line with our expectations, as each item x(i) will also be in the context of its respective neighbours and will therefore amplify its attractive force. A large number for k for example will force all points towards the centre of the canvas or if even larger, produce random scatter as the gradients amplify. In our experiments we use k = 10, for datasets with a few thousand samples, k should usually be below l. We also found, that the negative context is best with l = 20 for all sizes. Furthermore, we set both θ 1 = θ 2 = 1.0 for all experiments because the influence on selecting k, l is much larger. The graph context is also set to s = 10 (in our dataset the number of entities is close to the number of documents), the objective weight can be freely adjusted between around 0.8 ≤ θ 3 ≤ 1.2 to set the influence of the entity network. Similar to the semantic neighbourhoods in the first and second objective, the choice of s is significantly more influential than θ 3. A.4 LANDSCAPE VISUALISATIONS

Represent the short summary for finding the corresponding scientific article (long)

Dimensionality reduction algorithm to visualise text with network information, for example an email corpus or co-authorships.


Question: Represent the article for finding a single-sentence TLDR

Question: Machine learned models exhibit bias, often because the datasets used to train them are biased. This presents a serious problem for the deployment of such technology, as the ing models might perform poorly on populations that are minorities within the training set and ultimately present higher risks to them. We propose to use high-fidelity computer simulations to interrogate and diagnose biases within ML classifiers. We present a framework that leverages Bayesian parameter search to efficiently characterize the high dimensional feature space and more quickly identify weakness in performance. We apply our approach to an example domain, face detection, and show that it can be used to help identify demographic biases in commercial face application programming interfaces (APIs). Machine learned classifiers are becoming increasingly prevalent and important. Many systems contain components that leverage trained models for detecting or classifying patterns in data. Whether decisions are made entirely, or partially based on the output of these models, and regardless of the number of other components in the system, it is vital that their characteristics are well understood. However, the reality is that with many complex systems, such as deep neural networks, many of the "unknowns" are unknown and need to identified BID23. Imagine a model being deployed in law enforcement for facial recognition, such a system could encounter almost infinite scenarios; which of these scenarios will the classifier have a blind-spot for? We propose an approach for helping diagnose biases within such a system more efficiently. Many learned models exhibit bias as training datasets are limited in size and diversity BID34 BID33, or they reflect inherent human-biases BID7. It is difficult for researchers to collect vast datasets that feature equal representations of every key property. Collecting large corpora of training examples requires time, is often costly and is logistically challenging. Let us take facial analysis as an exemplar problem for computer vision systems. There are numerous companies that provide services of face detection and tracking, face recognition, facial attribute detection, and facial expression/action unit recognition (e.g., Microsoft (msf, 2018), Google (goo, 2018), Affectiva (; aff, 2018) ). However, studies have revealed systematic biases in of these systems BID6 BID5, with the error rate up to seven times larger on women than men. Such biases in performance are very problematic when deploying these algorithms in the real-world. Other studies have found that face recognition systems misidentify [color, gender (women), and age (younger)] at higher error rates BID22. Reduced performance of a classifier on minority groups can lead to both greater numbers of false positives (in a law enforcement domain this would lead to more frequent targeting) or greater numbers of false negatives (in a medical domain this would lead to missed diagnoses).Taking face detection as a specific example of a task that all the services mentioned above rely upon, demographic and environmental factors (e.g., gender, skin type, ethnicity, illumination) all influence the appearance of the face. Say we collected a large dataset of positive and negative examples of faces within images. Regardless of how large the dataset is, these examples may not be evenly distributed across each demographic group. This might mean that the ing classifier performs much less accurately on African-American people, because the training data featured few examples. A longitudinal study of police departments revealed that African-American individuals were more likely to be subject to face recognition searches than others BID15. To further complicate matters, even if one were to collect a dataset that balances the number of people with different skin types, it is highly unlikely that these examples would have similar characteristics across all other dimensions, such as lighting, position, pose, etc. Therefore, even the best efforts to collect balanced datasets are still likely to be flawed. The challenge then is to find a way of successfully characterizing the performance of the ing classifier across all these dimensions. The concept of fairness through awareness was presented by BID9, the principle being that in order to combat bias we need to be aware of the biases and why they occur. This idea has partly inspired proposals of standards for characterizing training datasets that inform consumers of their properties BID20. Such standards would be very valuable. However, while transparency is very important, it will not solve the fundamental problem of how to address the biases caused by poor representation. Nor will it help identify biases that might still occur even with models trained using carefully curated datasets. Attempts have been made to improve facial attribute detection by including gender and racial diversity. In one example, by BID29, were improved by scraping images from the web and learning facial representations from a held-out dataset with a uniform distribution across race and gender intersections. However, a drawback of this approach is that even images available from vast sources, such as Internet image search, may not be evenly balanced across all attributes and properties and the data collection and cleaning is still very time consuming. To address the problem of diagnosing bias in real world datasets we propose the use of high-fidelity simulations BID30 to interrogate models. Simulations allow for large volumes of diverse training examples to be generated and different parameter combinations to be systematically tested, something that is challenging with "found" data scrapped from the web or even curated datasets. Simulated data can be created in different ways. Generative adversarial networks (GANs) BID17 are becoming increasingly popular for synthesizing data BID31. For example, GANs could be used to synthesize images of faces at different ages BID40. However, GANs are inherently statistical models and are likely to contain some of the biases that the data used to train them contain. A GAN model trained with only a few examples of faces with darker skin tones will likely fail to produce a diverse set of high quality synthesized images with this attribute. Parameterized graphics models are an alternative for training and testing vision models BID36 BID15 BID35. Specifically, it has been proposed that graphics models be used for performance evaluation BID19. As an example, this approach has been used for models for pedestrian detection BID35. To the best of our knowledge graphics models have not been employed for detecting demographic biases within vision models. We believe that demographic biases in machine learned systems is significant enough a problem to warrant further attention. The contributions of this paper are to: present a simulated model for generating synthetic facial data, show how simulated data can be used to identify the limitations of existing face detection algorithms, and to present a sample efficient approach that reduces the number of simulations required. The simulated model used in this paper is made available. With more adoption of machine learned algorithms in real-world applications there is growing concern in society that these systems could discriminate between people unfairly 12. In cases where these systems provide a reliable signal that the output has a low confidence, these can be described as known unknowns. However, many learned models are being deployed while their performance in specific scenarios is still unknown, or their prediction confidence in scenarios is not well characterized. These can be described as unknown unknowns. Lakkaraju et al. BID23 published the first example of a method to help address the discovery of unknowns in predictive models. First, the search-space is partitioned into groups which can be given interpretable descriptions. Second, an explore-exploit strategy is used to navigate through these groups systematically based on the feedback from an oracle (e.g., a human labeler). Bansal and Weld proposed a new class of utility models that rewarded how well the discovered unknown unknowns help explain a sample distribution of expected queries BID3. We also employ an explore-exploit strategy in our work, but rather than rely on an oracle we leverage synthetic data. Biases often from unknowns within a system. Biases have been identified in real-world automated systems applied in domains from medicine BID43 to criminal justice BID2. While the exact nature of the problem is debated BID14 it is clear that further research is needed to address bias in machine learned systems and actionable remedies to help practitioners in their day to day work are necessary. Numerous papers have highlighted biases within data typically used for machine learning BID7 BID34 BID33 and machine learned classifiers BID5. Biases in datasets can be caused in several ways. BID34 identified selection bias, capture bias, and negative set bias as problems. " Selection bias" is related to the tendency for certain types, or classes, of images to be included in datasets in the first place. " Capture bias" is related to how the data are acquired and may be influenced by the collectors' preferences (e.g. in images of certain people, say celebrities, or points of view, lighting conditions, etc.). Taking faces as an example, these biases typically manifest as a greater proportion of images including white males, frontal head poses and smiles or neutral expressions. " Negative set bias" is related to the examples in the dataset of the "rest of the world". As an example, if the negative examples included in a dataset of faces are skewed, the model will learn a warped representation of what is not a face. However, even if one were to make every effort to address these three types of bias, the problem may still exist that artifacts created by humans contain the biases that human have without our explicit knowledge BID7. In our discussion, we will call this "human bias".With increasing frequency, data hungry machine learning classifiers are being trained on large-scale corpora collected from Internet sources BID8 BID11 ) and/or via crowdsourcing BID26. Such data collection makes curation challenging as there are a vast number of samples to label and sort. The Internet is a data source that is likely to be subject to "selection bias", "capture bias", "negative set bias" and "human bias". Furthermore, BID33 argue that more powerful feature descriptors (such as those generated by a deep convolutional neural network versus a simple hand crafted approach) may actually exacerbate the problem of bias and accentuate biases in the ing classifier. Even in the best case scenario, feature representations alone cannot intrinsically remove the negative bias problem. Thus there is still a need for approaches that help address the issues of bias within machine learning, and this problem is becoming more acute. To this end, we propose a practical approach to use high-fidelity simulations to diagnoses biases efficiently in machine vision classifiers. Face detection was one of the earliest applications of computer vision. Below we describe significant landmarks in the development of face detection algorithms. Earlier methods relied on rigid templates with boosting learning methods commonly employed. Hand-crafted features used for learning included Haar like features, local binary patterns and histograms of gradients. A major landmark was the Viola-Jones BID39 classifier based on Haar features. This work presented a fast and accurate method and was widely used thanks to implementations in OpenCV and other computing frameworks. Deep convolutional neural networks have since surpassed the performance of previous methods BID12 BID41 and can be used to learn face detection and attribute classification together in a single framework BID27. Face detection has matured into a technology that companies and agencies are deploying in many real world contexts BID42; these include safety critical applications. Application programming interfaces (APIs) and software development kits (SDKs) are two ways in which companies are exposing these models to other businesses and to consumers. All the APIs we study in this paper use convolutional neural networks for face detection, making diagnosing points of failure or bias challenging. The approach we present for identifying biases is algorithm independent, but we believe is particularly suited to models trained on large corpra and using powerful feature descriptors. Face detection is frequently the first step applied in screening images to include in datasets used for subsequent facial analysis tasks (e.g., face recognition or expression detection). If the face detector used for such tasks is biased, then the ing data set is also likely to be biased. This was found to be the case with the commonly used Labeled Faces in the Wild (LFW) dataset . One study found it to be 78% male and 85% White BID18. We believe a simulation-based approach, using an artificial human (see FIG0, could help characterize such biases and allow researchers to account for them in the datasets they collect. We propose to use simulation to help interrogate machine learned classifiers and diagnose biases. The key idea here is that we repeatedly synthesize examples via simulation that have the highest likelihood of breaking the learned model. By synthesizing such a set, we can then take a detailed look at the failed examples in order to understand the biases and the blind spots that were baked into the model in the first place. We leverage the considerable advancements in computer graphics to simulate highly realistic inputs. In many cases, face detection algorithms are designed to be able to detect faces that occupy as few as 36 x 36 pixels. The appearances of the computer generated faces are as close to photo-realistic in appearance as is necessary to test the face detectors, and at low resolutions we would argue that they are indistinguishable from photographs. While simulated environments allow us to explore a large number of parameter combinations quickly and systematically, varying lighting, head pose, skin type, etc. often require significant computational resources. Furthermore, exhaustive search over the set of simulations is rendered infeasible as each degree of freedom in simulation leads to exponentially increasing numbers of examples. Therefore, we need a learning method that can intelligently explore this parameter space in order to identify regions of high failure rates. In this work, we propose to apply Bayesian Optimization BID4 to perform this parameter search efficiently. Formally, lets denote θ as parameters that spawns an instance of simulation s(θ). This instance then is fed into the machine intelligence system in order to check whether the system correctly identifies s(θ). Consequently, we can define a composite function Loss(s(θ)), that captures the notion of how well the AI system handles the simulation instance generated when applying the parameters θ. Note that the function Loss(·) is similar to the loss functions used in training classifiers (e.g. 0-1 loss, hinge loss etc.). Our goal then is to find diverse instances of θ such that the composite loss function attains values higher than a set threshold. FIG1 graphically describes such a composition. Bayesian Optimization allows us to tackle this problem by first modeling the composite function Loss(s(θ)) as a Gaussian Process (GP) BID28. Modeling as a GP allows us to quantify The composite function, which takes as input the simulation parameters θ, in order to first produce a simulation s(θ). This in turn is fed into a machine classification system to compute a loss function characterizing whether the system performed correctly or poorly.uncertainty around the predictions, which in turn is used to efficiently explore the parameter space in order to identify the spots that satisfy the search criterion. In this work, we follow the recommendations in BID32, and model the composite function (0-1 loss) via a GP with a Radial Basis Function (RBF) kernel, and use Expected Improvement (EI) as an acquisition function. Within the AirSim BID30 ) environment we created an agent torso. The agent was placed in a room with multiple light sources (from windows, overhead lights and spotlights). Pictures of the avatar and the layout of the room are shown in FIG0. The avatar and environment are being made available. The skin type of the agent could be varied continuously from a lighter to darker skin type (allowing us to mimic different levels on the Fitzpatrick Classification Scale BID13), and aging of the skin could be customized via the textures. The agent's facial position, facial actions (e.g., mouth opening or eye lids closing) could be fully customized. In this paper, our goal was to illustrate how simulated data can be used to identify and evaluate biases within classifiers and not to exhaustively evaluate each face API's performance with respect to every facial expression or lighting configuration. However, our method could be used to do this. We manipulated the following parameters in order to evaluate how the appearance of the face impacted the success or failure of the face detection algorithms. The parameters were varied continuously within the bounds specified below. Angles are measured about the frontal position head position. For facial actions (mouth opening and eye lids closing) the mappings are specified in Facial Action Coding System BID10 intensities. We compared four facial analysis APIs in our experiments. Microsoft, Google, IBM and Face++ all offer services for face detection (and classification of other facial attributes). The APIs all accept HTTP POST requests with URLs of the images or binary image data as a parameter within the request. These APIs return JSON formatted data structures with the locations of the detected faces. Unfortunately, detailed descriptions of the data used to train each of the face detection models were not available. To the best of our knowledge, the models exposed through these APIs all use deep convolutional neural network architectures and are trained on millions of images. The documentation reports that a face is detectable when its size is 36 × 36 to 4096 × 4096 pixels and up to 64 faces can be returned for an image. Google: The documentation did not report minimum size or resolution requirements. The documentation reports that the minimum pixel density is 32 × 32 pixels per inch, and the maximum image size is 10 MB. IBM published a statement in response to the article by in which they further characterize the performance of their face detection algorithm.3Face++: The documentation reports that a face is detectable when its size is 48 × 48 to 4096 × 4096 pixels. The minimal height (or width) of a face should be also be no less than 1/48 of the short side of image. An unlimited number of faces can be returned per image.: Distribution of each simulation parameter for successfully detected faces and missed detections. The distribution for skin types and ages skews darker and older respectively for false negatives than for true positives. The Bayesian optimization reveals this difference with fewer simulations, such that given 1000 samples the differences are apparent with BO and not with random sampling. The skin tone ranges from light to dark and age ranges from young to old. For example images see FIG2 We used two approaches for searching our space of simulated faces for face detection failure cases. The first was randomly sampling parameters for generating face configurations and the second was using Bayesian optimization. We compared these sampling methods for the different face detection APIs. This type of analysis would have been difficult, if not impossible, without the ability to systematically synthesize data. FIG3 shows boxplots of the demographic parameters for faces correctly detected and faces missed. The are shown for the skin type on a normalized scale of 0 (lighter skin type, Fitzpatrick I) to 1 (darker skin type, Fitzpatrick VI) and age on a normalized scale of 0 (unwrinkled) to 1 (heavily wrinkled). From left to right illustrate the performance for the Microsoft, Face++, Google and IBM classifiers. FIG4 shows the cumulative number of failure cases identified with increasing numbers of simulations (faces generated). The are presented across all the APIs with the error bars representing the standard error. When using random sampling the sample (or simulation) efficiency for finding failure cases is significantly reduced compared to our Bayesian approach. After 800 samples 44% more failures (724 vs. 503) were found using Bayesian optimization. First, let us discuss the performance (characterized by true positives and false negatives) of the APIs with regard to the skin type and age appearance of the subjects. FIG3 shows that the detectors consistently failed more frequently on faces with darker skin types and to a lesser extent on faces with older appearances. The missed detections were heavily skewed towards the upper end of the skin type and age ranges. Overall, the IBM API skew was the least extreme. This is probably due to the fact that the detectors were improved 4 following the presented in BID5 identifying biases within previous versions. This shows that paying attention to the training data used to create the models behind these APIs can significantly mitigate the biases with the ing models. While our are only based on one base facial model, they provide that support prior work BID6. While our are only based on one base facial model, they provide evidence that supports prior work BID6. We would need further models with alternate bone structures to draw more about the generalizability of these , especially with regard to other demographic variables (e.g., gender). The age manipulation of the avatar was somewhat constrained as only the texture (and not the shape) of the face was varied. This is a limitation and more realistic age manipulation warrants future work. Using naive sampling techniques (e.g., random sampling or grid search), the cost of search is exponential with the number of parameters. Consider that there are three axes of head rotation, twentyeight facial action units, thousands of possible facial expressions, and millions of potential lighting configurations. Soon naive sampling techniques become impractical. We show that a Bayesian optimization approach can significantly reduce the number of simulations required to find an equal number of failure cases. In a simple experiment with six parameters, the Bayesian approach led to an over 40% improvement in efficiency with respect to finding the false negatives (missed detections). With a larger number of parameters this improvement will be much more dramatic. In addition, the improvement in sample efficiency was the most dramatic (over 500%) for the classifier with the fewest number of missed detections overall (IBM) (see FIG4 . This suggests that BO can further improve efficiency as failures become more challenging to find. Bias in machine learned systems can be introduced in a number of ways. In complex models these biases can be difficult to identify. Using data generated via a realistic synthetic environment we have been able to identify demographic biases in a set of commercially available face detection classifiers. These biases are consistent with previous on photo datasets BID5 . While the synthetic faces we used were not quite as realistic as photographs we believe that this empirical finding supports the use of parametric simulations for this problem. We present an approach that leverages highly-realistic computer simulations to interrogate and diagnose biases within ML classifiers. We propose the use of simulated data and Bayesian optimization to intelligently search the parameter space. We have shown that it is possible to identify limits in commercial face detection systems using synthetic data. We highlight bias in these existing classifiers which indicates they perform poorly on darker skin types and on older skin texture appearances. Our approach is easily extensible, given the amount of parameters (e.g., facial expressions and actions, lighting direction and intensity, number of faces, occlusions, head pose, age, gender, skin type) that can be systematically varied with simulations. We used one base facial model for our experimentation. This limits the generalization of our and the ability for us to determine whether the effects would be similar or different across genders and other demographic variables. Synthetic faces with alternate bone structures would need to be created to test these hypotheses. While the initial cost of creating the models is high, they can be used to generate large volumes of data, making synthetics cost effective in the long-run. Age modeling in face images should be improved using GAN or improved parametric synthetic models. A limitation of our work is that the aging was only represented via texture changes. We plan to investigate GAN-based approaches for synthesis and compare these to parametric synthesis. A hybrid of parametric and statistical models could be used to create a more controllable but diverse set of synthesized faces. Future work will consider retraining the models using synthetic data in order to examine whether this can be used to combat model bias.

We present a framework that leverages high-fidelity computer simulations to interrogate and diagnose biases within ML classifiers.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.


Question: represent the article for finding a single-sentence tldr

Question: Point clouds are a flexible and ubiquitous way to represent 3D objects with arbitrary resolution and precision. Previous work has shown that adapting encoder networks to match the semantics of their input point clouds can significantly improve their effectiveness over naive feedforward alternatives. However, the vast majority of work on point-cloud decoders are still based on fully-connected networks that map shape representations to a fixed number of output points. In this work, we investigate decoder architectures that more closely match the semantics of variable sized point clouds. Specifically, we study sample-based point-cloud decoders that map a shape representation to a point feature distribution, allowing an arbitrary number of sampled features to be transformed into individual output points. We develop three sample-based decoder architectures and compare their performance to each other and show their improved effectiveness over feedforward architectures. In addition, we investigate the learned distributions to gain insight into the output transformation. Our work is available as an extensible software platform to reproduce these and serve as a baseline for future work. Point clouds are an important data type for deep learning algorithms to support. They are commonly used to represent point samples of some underlying object. More generally, the points may be extended beyond 3D space to capture additional information about multi-sets of individual objects from some class. The key distinction between point clouds and the more typical tensor data types is that the information content is invariant to the ordering of points. This implies that the spatial relationships among points is not explicitly captured via the indexing structure of inputs and outputs. Thus, standard convolutional architectures, which leverage such indexing structure to support spatial generalization, are not directly applicable. A common approach to processing point clouds with deep networks is voxelization, where point clouds are represented by one or more occupancy-grid tensors (,). The grids encode the spatial dimensions of the points in the tensor indexing structure, which allows for the direct application of convolutional architectures. This voxelization approach, however, is not appropriate in many use cases. In particular, the size of the voxelized representation depends on the spatial extent of the point cloud relative to the spatial resolution needed to make the necessary spatial distinctions (such as distinguishing between different objects in LIDAR data). In many cases, the required resolution will be unknown or in enormous tensors, which can go beyond the practical space and time constraints of an application. This motivates the goal of developing architectures that support processing point cloud data directly, so that processing scales with the number of points rather than the required size of an occupancy grid. One naive approach, which scales linearly in the size of the point cloud, is to'flatten' the point cloud into an arbitrarily ordered list. The list can then be directly processed by standard convolutional or fully-connected (MLP) architectures directly. This approach, however, has at least two problems. First, the indexing order in the list carries no meaningful information, while the networks do not encode this as a prior. Thus, the networks must learn to generalize in a way that is invariant to ordering, which can be data inefficient. Second, in some applications, it is useful for point clouds to consist of varying numbers of points, while still representing the same underlying objects. However, the number of points that can be consumed by the naive feedforward architecture is fixed. PointNet and exhibit better performance over the MLP baseline with a smaller network by independently transforming each point into a high-dimensional representation with a single shared MLP that is identically applied to each individual point. This set of derived point features is then mapped to a single, fixed-sized dense shape representation using a symmetric reduction function. As such the architectures naturally scale to any number of input points and order invariance is built in as an architectural bias. As a , these architectures have been shown to yield significant advantages in applications in which point clouds are used as input, such as shape classification. The success of PointNet and DeepSet style architectures in this domain shows that designing a network architecture to match the semantics of a point cloud in a more efficient, and better performing network. Since point clouds are such a useful object representation, it's natural to ask how we should design networks to decode point clouds from some provided shape representation. This would allow for the construction of point cloud auto-encoders, which could serve a number of applications, such as anomaly detection and noise smoothing. Surprisingly, the dominant approach to designing such a differentiable point cloud decoder is to feed the dense representation of the desired object through a single feedforward MLP whose is then reshaped into the appropriate size for the desired point cloud. This approach has similar issues as the flat MLP approach to encoding point clouds; the decoder can only produce a fixed-sized point cloud while point clouds are capable of representing objects at low or high levels of detail; the decoder only learns a single deterministic mapping from a shape representation to a point cloud while we know that point clouds are inherently random samples of the underlying object. The primary goal and contribution of this paper is to study how to apply the same lessons learned from the PointNet encoder's semantic congruence with point clouds to a point cloud decoder design. As such, we build on PointNet's principles to present the'NoiseLearn' algorithm-a novel, simple, and effective point cloud decoding approach. The simplicity of the decoding architectures and the increase in performance are strong indicators that sample-based decoders should be considered as a default in future studies and systems. In addition, we investigate the operation of the decoders to gain insight into how the output point clouds are generated from a latent shape representation. Point cloud decoders are a relatively unexplored area of research. Among the works which describe an algorithm that produces a point cloud, the majority focus their efforts on learning a useful latent shape representation that is then passed to a MLP decoder. PU-Net is one such example, in which they design a novel point cloud upsampling network which uses a hierarchical approach to aggregating and expanding point features into a meaningful latent shape representation. To decode the learned shape representation into a point cloud, the latent vector is then passed through a feedforward MLP to produce a fixed number of points. This implies that the network would need to be retrained to allow for a different upsampling rate, which unlikely to be a desired property of an upsampling algorithm. TopNet recognizes the data inefficiency of using a single MLP to decode a point cloud and instead reorganizes their MLP into a hierarchical tree structure in which MLPs at the same level share the same parameters. Their show that addressing this inefficiency allows for better performance with a smaller parameter count. Similarly, in "Learning Localized Generative Models for 3D Point Clouds via Graph Convolution" augments their decoder by assuming a graph structure over the decoded point cloud and employing graph convolutions. However, despite improved performance neither approach addresses the other issues that come with using MLPs to decode entire point clouds, namely the fixed-size output. "Point Cloud GAN" and PointFlow take a different approach to producing a point set in a generative setting. Instead of learning a single mapping from any latent vector directly to its decoded point cloud, they learn a function parameterized by the latent vector which transforms low-dimensional Gaussian noise to a 3D point on the surface of the object described by the latent shape representation. This sampling based approach is more in line with the semantics of point clouds. First, an arbitrary number of points can be drawn from the Gaussian noise to produce a point cloud consisting of that number of points without requiring any changes to or retraining of the algorithm. Second, every individual point is decoded independently and identically, which avoids the data inefficiency issues that come with using MLPs to process set data. While this sampling approach has several desirable properties and appears promising, it's unclear whether the algorithm is applicable outside of the GAN settings these two papers inhabit, if they require specific bespoke loss functions to be trained effectively, or if they are capable of outperforming the baseline MLP approach according to other metrics. A point cloud is a set of n 3D points C = {p 1, . . ., p N}, where each p i ∈ R 3. In general, each p i may have additional auxiliary information associated with it via non-spatial dimensions. While all of our architectures easily generalize to include such information, in this paper, we focus on point clouds that exclusively encode shapes without auxiliary information. A point cloud auto-encoder takes a point cloud C with n points and outputs a point cloudĈ with m points that is intended to represent the shape described by C. While often n = m, we are interested in the general case when n and m may be different, which corresponds to up-scaling or downscaling C. Each auto-encoder will be comprised of an encoder E(C), which takes an input point cloud and outputs a latent shape representation h in R l, and a decoder D(h) which maps a latent representation to an output point cloud of the appropriate size. Thus, given an input point cloud C, the auto-encoder output is given byĈ = D(E(C)). In this paper, we focus on the Chamfer distance as the measure of auto-encoder quality. Intuitively this loss function measures how wellĈ matches C in terms of the nearest neighbor inĈ to each point in C and vice versa. Specifically, if dist(p,Ĉ) gives the distance between point p and the nearest neighbor in point cloudĈ, our loss function is defined by Since the focus of this paper is on point-cloud decoders, all of our architectures use the same pointcloud encoder architecture, while varying the decoder architecture. Below, we first overview the common PointNet-style encoder used followed by a description of the four decoders considered in our experimental analysis, which include three sample-based decoders. PointNet handles unordered input by recognizing that a symmetric function g (such as element-wise max or sum) produces the same regardless of the order of its inputs. PointNet thus learns a single transformation function f that maps individual points to an l-dimensional representation and then combines those representations via g. That is, the latent encoding produced by PointNet for a point cloud C = {p 1, . . ., p n} is the l dimensional vector As desired, E(C) is invariant to the ordering of points in C and applies to any number of points. We learn an MLP representation of f, with input space R 3, encoding points, and output space R l, encoding the latent representation or point feature. We use max as the reduction function g to map the arbitrary number of ing point features to a single fixed-size latent shape representation. The hidden layers and size of the latent shape representation for each instantiation of this encoder architecture can be found in Table 1. Most prior work has used MLP decoders, which we consider here as a baseline approach. An MLP decoder is a fully connected network that takes the latent shape representation as input and outputs an m × 3 output vector, which represents the m output points. Accordingly, MLP decoders are parameterized by the number and size of their fully connected layers. In our experiments, each fully connected layer consists of parameterized ReLU units with a batch normalization layer. Our main focus is on sample-based decoders, which allow for an arbitrary number of outputs points to be produced for a latent shape representation. In particular, given a latent shape representation h, each of our decoders is defined in terms of a point feature sampling distribution S(h), where the decoder produces a point-cloud output by sampling m point features from S(h). Once we have a set of M independently sampled point features from our sampling distribution S(h) we need to transform each one into a triple representing that point's location. Note that we are now in an identical but opposite situation as the point cloud encoder. Whereas the encoder had to transform independent point samples of some underlying object into corresponding high-dimensional representations, our decoder now has to transform independently sampled high-dimensional point representations into a point in space on the surface of the target object. Therefore, we can simply apply the same style of PointNet encoding mechanism with different input and output tensor sizes to implement an effective point feature decoder. The sizes of the hidden layers in our decoder network can be seen in Table 1. By applying the shared MLP point decoder to each sampled point feature, we can directly decode point clouds of arbitrary size. Below we describe three architectures for S, which are compared to each other and the baseline MLP decoder in our experiments. NoiseAppend Decoder. NoiseAppend is similar to the sampling approach described in "Point Cloud GAN" by. They sample point features by simply sampling from a multivariate Gaussian distribution with zero mean and unit variance before appending the sampled noise to the latent shape vector. That is, S(h) = concat (h, N (0, I)). However, this requires us to decide how many elements of noise should be appended to the latent shape representation. state that the size of the appended noise vector should be'much smaller than' the size of the latent shape representation, but it's not clear how much noise is necessary to allow the decoder to fully represent the shape. Ultimately this is an additional hyperparameter that needs to be investigated and tuned. NoiseAdd Decoder. NoiseAdd builds on the concept of adding unit Gaussian noise to the latent shape vector with the goal of avoiding the additional hyperparameter that NoiseAppend introduces. This can be easily accomplished by treating the entire latent vector as the mean of a Gaussian distribution with unit variance. That is, S(h) = N (h, I). However, this violates the claim by that the amount of noise introduced to the ing point feature samples should be much smaller than the size of the latent shape representation itself. Therefore, it may be the case that uniformly adding noise to every element of the latent vector obscures the crucial information it represents. NoiseLearn Decoder. NoiseLearn attempts to instead learn a small separate function V (h) which predicts the log-variance of the desired point feature distribution. Specifically, S(h) = N h, e V (h)/2 I. We define V (h) as a small MLP, the size of which can be seen in Table 1. By allowing the network to choose the amount and location of noise to be added to the latent shape vector, we hope that it will learn both to add an appropriate amount of noise for the target shape while Figure 2: Diagrams of the different approaches to deriving a distribution from the latent shape representation h. conserving the information necessary to accurately reconstruct it without introducing any additional hyperparameters. We evaluated each decoding architecture by training several instantiations of each architecture on a point cloud auto-encoding problem derived from the ModelNet40 dataset, which consists of over 12,000 3D models of 40 different common object classes. The dataset has a prescribed train/test split, with approximately 9800 models in the training dataset and 2500 in the test dataset. We randomly select 10% of the training data to use for validation during training. Before training, each object model in the ModelNet40 dataset is used to generate a uniformlysampled point cloud with 4096 points which is then scaled to fit within the unit sphere. For all autoencoder network models, at each iteration of training, the point clouds are randomly downsampled to 1024 points before being used to update the network parameters. The helps reduce the computational cost of training and also encouraging better generalization. During training, each decoded point cloud consists of 1024 points. We use the Chamfer distance as the loss function due to its relative speed and capability to directly compare point clouds of unequal sizes without modification. Each network is trained for 100 epochs using the ADAM optimizer with an initial learning rate of 10 −3, where each epoch performs a parameter update on each training example. The learning rate is decreased by a factor of 10 at epoch 50 and epoch 80. We trained five instantiations of each of the four network architectures with each instantiation varying the number of parameters as shown in Table 1 (note that we were not able to scale down the MLP for the smallest parameter setting). For each instantiation we ran the entire training process 15 times and all show average performance across the 15 runs. All code and infrastructure for "push-button" replication of our experiments open-source (Github/Gitlab location removed for anonymous review -code will be privately made available to reviewers through a comment approximately a week after submission). Quantitative Results. Figure 3 shows the validation loss along the learning curves for the 2M parameter instantiation of each architecture. The relative ordering of the architectures is consistent after the initial phase of training, with all curves flattening out by 100 epochs. First, the large jumps in the MLP training (due to unstable training runs) show that it was much less stable to training compared to the sample-based architectures. While effort was spent trying to specialize the training configuration for the MLP, stability remained an issue. 1 In contrast the runs for each sample based architecture were stable and similar. Ignoring the MLP stability, it performs similarly to worst performing sample-based architectureby the end of the learning curve. The three sample based architectures show rapid progress early in learning and then settle into a consistent ordering with NoiseLearn performing best, followed by NoiseAppend, and then NoiseAdd. This suggests that NoiseAdd's approach of adding uniform noise to the latent representation may be obscuring information needed for accurate reconstruction, compared to NoiseAppend, which separates noise from the shape representation. On the other hand, we see that while NoiseLearn also adds noise to the latent representation, it is able to outperform NoiseAppend. This indicates the importance of being able to intelligently select how much noise to add to different components of the representation. Apparently, this allows NoiseLearn to avoid obscuring critical information in the latent representation needed for accurate reconstruction. Figure 4 shows the average test set performance after 100 epochs of training of each size instantiation of the four architectures (note the log scale). The appendix also shows more detailed broken down for each of 5 selected object classes. The three sample based architectures show relatively consistent improvement in performance as the sizes grow by orders of magnitude. Rather, the MLP shows initial improvement, but then performance decreases significantly past 100K parameters. We looked further into the behavior of the MLP architecture for the larger parameter sets. We observed that the larger MLPs showed a similar decrease in performance on even the training data, indicating that the problem is not necessarily overfitting but also difficulty of the optimization. It is possible that with substantially more epochs the MLP performance would improve, but at great cost. This indicates that the MLP architecture is much less efficient at exploiting larger network sizes than the more structured sample-based architectures. It is possible that the architecture and training hyperparameters could be tweaked to improve the large MLP networks' performance, such as by adding additional regularization via weight decay or other mechanisms. However, we consider this tweaking to be outside the scope of this work, and note that none of the sampling based architectures required any such tweaking to achieve competitive performance at all parameter counts. NoiseParam-2m NoiseAdd-2m NoiseAppend-2m Figure 5: Examples of networks' auto-encoding on several previously unseen objects. Overall the give strong evidence that the sample-based architectures encode a bias that is much better matched to decoding the class of shapes in ModelNet40 compared to MLPs. The structured sample-based architectures, compared to the MLP, in more stable learning and the ability to continually improve as the architectures grow in size. Further, we see that the NoiseLearn architecture, which avoids the need to specify hyperparameters to control the amount of noise performs the best, or near best, across all network sizes and number of epochs. Illustrative Qualitative Results. Figure 5 shows each network's performance three test objects not seen in the training data. The point cloud decoded by the MLP network appears to be more evenly distributed spatially, while the sampling-based approaches are better able to capture finer detail in the target shape, such as the stool's thin legs and crossbars. Among the sample-based approaches, no single approach is clearly dominant in terms of visual quality across the objects. It is interesting that all of the sample-based architectures tend to miss the same type of object details, e.g. the jets on the plane or the leg cross bars on the chair, which may be due to limitations of the PointNet encoders sized and/or architecture. Nevertheless, it is quite interesting that a single relatively small latent vector representation is able to encode the level of detail exibited in these . Each sampling architecture defines a function from the latent shape representation to a point feature distribution. The underlying latent representation inherently defines the manifold of the encoded shape. Rather, the injected noise (either via appending or addition) can be viewed as playing the role of indexing locations on the manifold for the generated point. Effectively, the primary difference between the sample-based architectures is how they use the noise to index locations and traverse the manifolds. Below we aim to better understand this relationship between noise and spatial indexing and how the architectures differ in that respect. In Figure 6 we demonstrate how each architecture uses noise by controlling the variance introduced to a trained network in two different ways. To examine how the decoder's output is influenced by individual elements of noise we show the output of these networks when all but one of the noise elements is held at a constant near-zero value. In the lower plots, we show the decoder's behavior when it only receives the union of the noise elements above. This demonstrates both how the network learns to exploit individual elements of noise and how the decoder combines those elements to produce a point cloud that spans the entire shape. For NoiseAppend all of the noise is of equal magnitude, so we just examine the first five elements of noise in its noise vector. NoiseLearn predicts individual variances for each element in the dense shape encoding, enabling us to select the five elements of noise with the highest variance, and therefore presumably the biggest contribution to the decoded point cloud. The appendix contains additional examples of noise manipulation. The plots shown in Figure 6 give us some insight into how the networks use noise to complete the decoded shape. Each individual element of noise appears to correspond to a learned path along the surface of the learned shape. The final point cloud then seems to be produced by'extruding' along those paths. NoiseLearn's use of only four significant elements of noise suggests that in this domain only three or four elements of noise is sufficient to achieve good coverage of the target shape. Figure 8 shows how individual noise channels change when the NoiseAppend architecture is modified to only append one, two, and three noise elements. With only one element of noise, we can see that the network effectively has to learn a single path that spans as much of the target shape as possible. With two elements of noise, the network instead seems to learn individual'loops' around the object which are transformed and rotated as necessary. Once the network has access to three elements of noise, we see the same behavior as the functional networks of learning small paths on the object's surface. If too little noise can seriously hurt NoiseLearn's performance, does adding too much noise do the same? Figure 7 shows the NoiseAppend architecture trained with different amounts of added noise to see if the same performance dropoff is present at both extremes. It appears that even when the noise vector is much larger than the dense shape representation, the decoder's overall performance is not impacted. However, note that adding large amounts of noise does significantly increase the parameter count, so there is a nontrivial cost to doing this. In this work, we evaluated and compared several realizations of a sample-based point cloud decoder architecture. We show that these sampling approaches are competitive with or outperform the MLP approach while using fewer parameters and providing better functionality. These advantages over the baseline suggest that sample based point cloud decoders should be the default approach when a network needs to produce independent point samples of some underlying function or object. To further this this area of research, we provide a complete open-source implementation of our tools used to train and evaluate these networks. The tables below show each architecture's average loss on each individual class in the ModelNet40 dataset. The best-performing network is bolded for each object class.

Represent the TLDR for finding the corresponding scientific article.

We present and evaluate sampling-based point cloud decoders that outperform the baseline MLP approach by better matching the semantics of point clouds.

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models