\subsection{New Intro}



Over 20 years ago, Pirolli and Card coined the term ``information foraging'' to denote the process of seeking and gathering information to apply towards a task\cite{pirolli1999information, pirolli2005sensemaking}.  While they focused on foraging for additional entities or rows of a dataset, foraging for additional data attributes can unlock new analytical capabilities.  For example, if a dataset contains a list of countries, adding the population of those countries as \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend enables per-capita analyses. 

Traditionally, crafting new attributes and augmenting a dataset with them is done prior to any visual analysis.  Incorporating this process into visual analytics workflows can benefit both the augmentation process as well as the underlying tasks of the system in several ways.  First, the need for an additional \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend may only arise based on insights generated during analysis; if data augmentation is embedded in the system, users can toggle back and forth between analysis and augmentation.  Second, the process of discovering data and crafting a new attribute is an analytical task in its own right, and typically consists of complex querying that can be abstracted away by a visual interface.  Lastly, user-driven curation of the \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of the dataset can serve as an additional medium of communication between user and system: the user can communicate domain knowledge by adding new \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend , and can likewise learn from the attributes that are discovered by the system.

However, there are many complications in integrating data augmentation into visual analytics systems.  Finding external data and matching it to the entites in the user's dataset is nontrivial.  Once connected to external data, it can be difficult to determine which data is relevant or useful to the user's analysis.  Some attributes may require the aggregation of multiple pieces of external data and can require joining through several intermediate entities.  These hurdles should be abstracted away so that the user is not required to be a database expert.  

Large scale information repositories provide the potential for interactive data augmentation because they put the potential data at the user's fingertips. However, in order to use such repositories a number of challenges must be addressed. If the repository is in the form of a data pool (a large collection of tables, e.g. WikiTables\cite{bhagavatula2013methodswikitable}), it can be difficult to determine which tables are relevant and joinable to the user's dataset. Tabular data can often be fragmented as well: for example, if a user is foraging for data on water usage across the United States, each municipality might be responsible for publishing its own data, and the formats may not align, resulting in sparse, error-prone joins.  Joining together multiple tables can also result in ambiguities over the type of join (left, right, outer, inner) that are difficult to resolve, and limit the types of attributes that can be constructed without additional data munging.  In addition, plenty of external data may be irrelevant or even harmful to the analysis tasks at hand.  
While there have been great strides in addressing some of the technical issues in using Data Lakes (see section~\ref{sec:related_work:data_augmentation}), little work has been done to allow a user to explore potential attributes and construct new ones.  Knowledge graphs offer an alternative to data lakes \DIFdelbegin \DIFdel{, }\DIFdelend because they simplify entity matching and joining ambiguities due to their graph format.  They bring different challenges, however, in scalability and complexity, and there is still a dearth of tools available to facilitate foraging over knowledge graphs.

In this paper, we present Auger\xspace, a visual analytics system for \DIFdelbegin \DIFdel{augmenting datasets with carefully crafted columns.  These columns }\DIFdelend \DIFaddbegin \DIFadd{Columnar data Augmentation through Visual Analytics.  Carefully crafted attributes }\DIFaddend are retrieved by running queries over knowledge graphs, as illustrated in Figure~\ref{fig:teaser}\DIFaddbegin \DIFadd{, before being added to a dataset as additional columns for downstream analysis}\DIFaddend .  First, each row of a dataset is mapped to an entity in a knowledge graph. Then, to help the user identify potential information of interest, Auger\xspace explores the local graph neighborhood about entities in the dataset to determine commonly held attributes.  Using visualizations of data quality, distribution, and the local topology of the knowledge graph, Auger\xspace guides the user through crafting additional \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of data without any programming or explicit querying.  Users can express complicated operations over the knowledge graph by interacting with these visualizations. In sum, Auger\xspace abstracts away the complexity of searching, retrieving, and joining data so that the user can focus on the exploratory task of determining which attributes are relevant to their analysis.



Auger\xspace shows the promise of integrating column augmentation as a foraging process within a VA system.  \DIFaddbegin \DIFadd{The design process presented in this work in section~\ref{sec:system} can be useful for future visual analytics tools to adopt columnar data augmentation as part of their workflow.  }\DIFaddend We claim that a resulting VA system can make the exploration and discovery of additional attributes interactive, and let users construct complex \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend without programming.  Users are able to discover new attributes as well as create attributes they had in mind.  The construction of the augmented dataset can improve the outcome of downstream analytical tasks, such as insight generation,  predictive modeling, or any other number of tasks that rely on the presence of a robust dataset.  

We offer evidence of these claims by describing two \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend for insight generation and predictive modeling.  In the first \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate how a domain scientist's analysis can be enriched through cycles of in-situ column augmentation and analysis.  And in the second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we show how a user can craft attributes that result in significantly more accurate machine learning models.  We also conduct a preliminary user study on two datasets to assess the usability of Auger\xspace.  The results confirm that users are able to both discover and craft relevant attributes easily and quickly. 


























In this work, we present the following contributions:
\begin{itemize}[topsep=3pt, itemsep=0pt,parsep=3pt]
    \item We present a visual analytics system, Auger\xspace, for exploratory data augmentation using knowledge graphs.  \DIFaddbegin \DIFadd{We also describe the design process of using visualization as a medium for query construction and knowledge graph exploration. 
    }\DIFaddend \item We provide \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend of Auger\xspace being applied to both insight generation and predictive modeling to demonstrate the generality of our approach.    
    \item We conduct a \DIFaddbegin \DIFadd{preliminary }\DIFaddend user study to assess the \DIFdelbegin \DIFdel{ability }\DIFdelend \DIFaddbegin \DIFadd{usability }\DIFaddend of our system \DIFdelbegin \DIFdel{to join both }\DIFdelend \DIFaddbegin \DIFadd{in joining }\DIFaddend semantically meaningful external data \DIFdelbegin \DIFdel{as well as data that improves a predictive model trained on the dataset}\DIFdelend \DIFaddbegin \DIFadd{across two different tasks, offering validation of our design}\DIFaddend .
\end{itemize}





\section{Background and Related Work}\label{sec:related_work}

\subsection{Knowledge Graphs}
Relational databases are one of the most popular formats to store data because they can be implemented and queried effectively and efficiently.
However, recently knowledge graphs have gained popularity for storing very large datasets because their entity-based model is conducive to how humans think about data.
Knowledge graphs represent data entities as nodes in a graph and the edges between those nodes are relationships between those entities.
The term \textit{knowledge graph} has been loosely used to describe collections of information.
Several different definitions have been offered in recent years.
In practice the term has been used interchangeably with \textit{knowledge base} or \textit{ontology}\cite{ehrlinger2016towards}.
Structuring entities in the form of an ontology is also used to organize conceptual spaces, for example for the evaluation of visual analytics systems\cite{chen2019ontological} and visualizations that support machine learning tasks\cite{sacha2018vis4ml}.

The current popularity of knowledge graphs to store information can be traced back to 2012, when Google introduced its \emph{Knowledge Graph}.
Google's Knowledge Graph is a repository that retrieves facts about entities in search terms on their search results pages\cite{googleknowledgegraph2012, Dong2014}, and is used to power the Google Assistant\cite{googleAssistant} and Google Home voice queries\cite{googleHome}. Recent work has demonstrated that the approach can generalize to other artificial intelligence tasks such as image captioning and conversational agents\cite{huang2019knowledge,zheng2018question,hixon2015learning}.

Since 2012, Wikipedia and other Wikimedia projects have collected information into the \textit{Wikidata} knowledge graph.
It contains broad information about tens of millions of entities.
Other large public repositories exist, from DBPedia\cite{auer2007dbpedia}, a collection of data resources, to the domain-specific universal protein knowledgebase \textit{UniProt}\cite{uniprot2017uniprot} and the linguistic resource \textit{WordNet}\cite{Miller1995}.
Many of these graphs are part of the Linked Open Data Cloud\cite{linkedopendata2020}, a knowledge graph that contains information about other knowledge graphs.
All these graphs contain a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

\DIFaddbegin \shah{knowledge graphs have also been used and investigated in many other research areas such as information retrieval, Natural Language Processing (NLP), and text analysis. For example, a number of approaches (e.g.,~\cite{Dalton2014, liu2015, Rao2013, Raviv2016}) and tools (e.g.,~\cite{Ferragina2010, Gabrilovich2013}) focused on text-centric information retrieval using entity links in underlying knowledge graphs. The information retrieval field has been advanced in many other aspects in knowledge graphs, e.g., for the discovery of emerging entities~\cite{Hoffart2014}, for structuring contents into topics~cite{Balasubramanian2010}, extracting relations~\cite{Mintz2009}, semantic search~\cite{Berant2013, Krishnan2018, Nanni2018}, for predicting the missing relations between entities~\cite{Dettmers2018, Yang2015}, etc. Researchers in NLP and text analysis fields have also utilized knowledge graphs from different perspectives, e.g., for embedding entities and words into a continuous vector space~\cite{Wang2014}, for extracting relation facts form text~\cite{Riedel2010, Hoffmann2011}, for the generation of questions and answers pairs automatically~\cite{Reddy2017}, for parsing and interpretation of user natural language semantically~\cite{Hakkani2014}, etc.}

\shah{Rephrasing of the below paragraph: In the case of our Auger\xspace system, users can tap knowledge repositories as a data source to augment a dataset through interactive visual support. Knowledge graphs have the advantage of providing clean, curated data which means that the need for data cleaning and matching is greatly reduced.
In addition, the entity-focused way in which knowledge graphs store information can make it easier to think about the relationship of data objects and thus helps users guide the augmentation process.}
\DIFaddend A lot of information is available in machine-readable form in knowledge graphs.
With Auger\xspace users can tap these knowledge repositories as a data source to augment a dataset.
Knowledge graphs have the advantage of providing clean, curated data which means that the need for data cleaning and matching is greatly reduced.
In addition, the entity-focused way in which knowledge graphs store information can make it easier to think about the relationship of data objects and thus helps users guide the augmentation process. \DIFaddbegin \DIFadd{Also, we believe that such visual analytics support as }Auger\xspace \DIFadd{could also be useful for researchers in other fields working with knowledge graphs. 
}\DIFaddend 



\subsection{Data Augmentation}
\label{sec:related_work:data_augmentation}
\DIFaddbegin \shah{comment: are we going to use a new term than data augmentation? if so then we need to change this section heading and this term in the below paragraphs}
\DIFadd{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new attributes (columns) for the objects.
For image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.
}

\DIFadd{Approaches to add columns to a dataset typically vary in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new attributes from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}

\DIFaddend Automated augmentation of datasets is of interest to the database and the data science communities.
In the database community, the goal of data augmentation can manifest in many ways. 
For example, before augmentation can take place, the first challenge is to find datasets that are suitable for joining. 
In this scenario, sometimes referred to as a ``data lake''\cite{Miller2018}, the data joining system assumes that there is a finite number of candidate datasets, and the task is to identify which of them can be joined with a user's base dataset.
Systems such as Google Goods\cite{halevy2016goods}, Infogather\cite{Yakout2012}, Octopus\cite{cafarella2009data}\DIFaddbegin \DIFadd{, }\DIFaddend Aurum\cite{fernandez2018aurum}, and work by Sarma et al.\cite{das2012finding} examine the attributes of the candidate datasets and learn the relationship between those attributes and the attributes of the input data.

Another type of data augmentation appears in the form of entity matching. 
Entity matching refers to finding the same entity in different datasets, often using machine learning techniques. Once identified, the entity can be removed (for deduplication) or merged (for augmentation). Examples of entity matching systems include Magellan\cite{konda2016magellan}, Nadeef\cite{dallachiesa2013nadeef}, Autojoin\cite{zhu2017auto}, and work by Mudgal et al.\cite{mudgal2018deep}.

What the data joining and entity matching approaches have in common is that the systems assume little knowledge about the data. The challenge is therefore to identify the commonalities between the datasets (e.g. schema matching) to determine if the datasets or the entities within are related and therefore joinable.
In Auger\xspace, we take a different approach.
Instead of assuming little to no knowledge about the candidate data, we use a knowledge graph as the source of ``candidate data.'' 
Although knowledge graphs can be loosely structured, they contain well-defined relations between entities. 
This property of knowledge graphs allows Auger\xspace find relevant data for augmentation with certainty and ease.
\DIFaddbegin \shah{Also, it is important to note that in Auger\xspace we focus on adding new attributes through this approach rather than adding just new instances while having the same set of attributes}
\DIFaddend 





\DIFdelbegin \DIFdel{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new features (columns) for the objects.
For image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.


\DIFdelend
\DIFdelbegin \DIFdel{Approaches to add columns to a dataset typically differ in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new features from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}\DIFdelend 


While automated approaches can discover joinable attributes for a given dataset, they lack semantic knowledge of expert users.
This may result in a large number of attributes that are irrelevant to the analysis problem being added to the dataset, hindering the users in further analyzing and gaining insights from the dataset.
In addition, in common one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.
Auger\xspace overcomes these issues by involving a user in the process.  Attributes are only added to the dataset if the user decides that they are helpful for their analysis.  

\begin{comment}
Consider a geospatial dataset, perhaps of temperature readings, with a column representing the country in which the reading took place.  In a broad data repository such as Wikidata, over 1200 attributes are potentially joinable to countries in just a single hop on the knowledge graph.  Joining irrelevant attributes into a dataset can complicate the downstream analytical tasks because of the curse of dimensionality.  

Even if an automated approach were to only select a manageable number of attributes to join, there may be semantic constraints that render certain attributes unusable that only a user would know about.  For example, consider an analyst who is studying housing prices in the United States.  If their task is to build a model for mortgage approvals, there may be certain attributes such as race that are considered protected characteristics that legally cannot be included in the analysis.  On the other hand, they may intentionally want to include these attributes so that they can do a post-hoc fairness analysis to see if those attributes can be predicted from the rest of the dataset~\cite{chouldechova2018frontiers}.  Using automated tools can be dangerous in many analysis environments, and we posit that a human in the loop tool like Auger\xspace mitigates those dangers. It is possible that the performance of Auger\xspace could be improved by using the data discovery system of Aurum rather than Wikidata or making use of the efficient queries from RDFFrames.  In addition, in one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.  For more details on how this is handled in Auger\xspace, see section~\ref{sec:system}.
\end{comment}



\subsection{Interactive Visual Data Curation}
\DIFaddbegin \shah{comment: We need to change this subsection heading.. May be merge with the previous one or ''Visual Analytics Support for Data Augmentation" or the name that we are going to use instead of data augmentation }

\DIFaddend Curating, improving, and augmenting an existing dataset is typically done to aid interpretation and sensemaking\cite{cramer2017impact} during analysis.
Sometimes referred to as ``data blending'', many commercial visualization systems, such as Tableau\cite{Tableau2020}, Alteryx\cite{Alteryx2020}, and D:Swarm\cite{DSwarm2020} support it by allowing users to join data tables.
The Google Data Studio\cite{GoogleDataStudio2020} and OpenRefine\cite{OpenRefine2020}, both based on the Webtables project\cite{Cafarella2008}, also support querying web APIs
and integrating data from them into a table.
In contrast to our approach, none of these tools supports users with discovering new data for augmentation in a systematic way.
Users have to find additional data on their own before they can join it to the base dataset.

\DIFaddbegin \shah{Other visual interactive approaches allow users to use and integrate or query data from heterogeneous web or local sources.
This includes helping users extract information from textual web sources\cite{Hoeber2017}, query knowledge graphs\cite{Hoefler2014}, query large metadata-rich heterogenous data stores~\cite{Smith2006}, identify relations at attribute levels in mixed data sets~\cite{Jurgen2014}, analyse associated categories in large categorial data tables~\cite{ Alsallakh2012}, or automatically create visual representations of information stored in knowledge graphs.}
\DIFaddend Other visual interactive approaches allow users to use and integrate data from heterogeneous web sources.
This includes helping users extract information from textual web sources\cite{Hoeber2017}, query knowledge graphs\cite{Hoefler2014}, or automatically create visual representations of information stored in knowledge graphs.
VAiRoma helps users extract and combine information from Wikipedia articles to create visualizations that provide insight into historical events\cite{Cho2015}.
Vispedia\cite{Chan2008} lets users interactively collect and integrate data from Wikipedia tables to create visualizations and answer analysis questions.
\DIFaddbegin \shah{Atlasify enables users to relate the query concept, corresponds to a Wikipedia article, to spatial entity in the underlying reference system, e.g., countries, political figure~\cite{Hecht2012}.}
\DIFaddend All of these approaches help users access information stored in different forms, but compared to Auger\xspace do not support the process of augmenting an existing dataset with additional data.

Tools for data wrangling, such as Wrangler, help users to interactively identify data quality problems, and fix them to improve their datasets\cite{Kandel2011wrangler}.
Data wrangling is an important step towards integrating data from diverse sources\cite{Stonebraker2013}.  In Auger\xspace, we leave issues of data wrangling to future research, and focus on the pressing need for the discovery of relevant attributes and their extraction from knowledge graphs.  The resulting datasets could then be fed into a tool like Wrangler.

Data improvement and curation tasks are also supported as part of creating and analyzing machine learning models.
Some approaches directly allow users to add and modify data by providing labels or corrections for train or test data\cite{Heimerl2012, Legg2019, mansoor2019delfi}.
Other systems support identifying mislabeled\cite{Ren2016, Alsallakh2017, strobelt2018s} or generally low quality instances\cite{Muehlbacher2013, Gleicher2020} for a range of different model types.
In contrast, Auger\xspace does not help identifying single instances to modify, but helps users adding relevant high-quality data columns to the entire dataset.

\begin{comment}
ML support in fixing data and labels:
- Helping users with correcting recorded and user labeled data~\cite{mansoor2019delfi}
- Boxer is relevant here, too (can help to find mislabeled instances) \cite{Gleicher2020}
- Also Heimerl~et~al.~\cite{Heimerl2012} active learning is a way to effectively label datasets (adding a new column to data items). (also AL: \cite{Legg2019})
- Squares\cite{Ren2016} helps users find mislabled instances in ML (classification) models, but does not support relabeling them directly. One use case shows how mislabeled instances ca be identified. Alsallakh~\cite{Alsallakh2017} claims that as well (but no use case).
Lot of work on finding mislabeled data.  Off the top of my head, Seq2Seq vis has debugging in the title. \cite{strobelt2018s}.
- M\"uhlbacher and Piringer~\cite{Muehlbacher2013} allow users to filter low quality instances to increase model quality.
\end{comment}





















\section{Knowledge Graphs}

\label{sec:knowledge_graphs}



\begin{comment}
Knowledge graphs are a generic data format in which entities are represented as nodes on a graph and the edges between those nodes represent relationships between those entities. They are used because of their flexibility and the ease of reading and writing information about entities.  Much of the current concept of knowledge graphs can be traced back to 2012, when Google introduced its Knowledge Graph, an information repository which retrieves facts about entities in search terms on their search results pages~\cite{googleknowledgegraph2012}.  Since then, Wikipedia and other Wikimedia projects have collected information into \textit{Wikidata}, a knowledge graph with broad information about tens of millions of entities.  Other large public repositories exist ranging from the collection of datasets in \textit{DBpedia}~\cite{auer2007dbpedia} to the universal protein knowledgebase \textit{UniProt}~\cite{uniprot2017uniprot}, and in fact, many of these knowledge graphs are linked themselves in the Linked Open Data Cloud~\cite{linkedopendata2020}.  These knowledge graphs represent a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

The term \textit{knowledge graph} has been loosely used to describe collections of information.   Several different definitions have been offered in recent years, and in practice the term has been used interchangably with \textit{knowledge base} or \textit{ontology}~\cite{ehrlinger2016towards}.
\end{comment}

We define our usage of the term, knowledge graph, in this section, and describe the sufficient characteristics of a knowledge graph that we assume for our system. We assume that knowledge graphs contain entities as nodes and express attributes of those entities as edges, connecting the source entity to either another entity (to express a type of relationship) or to a literal (such as a string, number or datetime).  The number of edges connected to a node can be a proxy for the number of attributes that are held by that entity.  The flexibility of this structure allows for complex relationships to be expressed between entities, including one-to-one (i.e. a country has one head of state), one-to-many (a country is composed of multiple municipalities), and many-to-many (countries share borders with other countries non-exclusively).


Auger\xspace requires that the connected knowledge graph has a data retrieval endpoint that can respond to simple queries about entities and their neighbors.  The connected graph must also have some sort of service to map from the values in the dataset uploaded to Auger\xspace to the entities on that knowledge graph.  For example, if a dataset with U.S. states is uploaded, there must be an existing service that maps from ``New York'' or ``NY'' to the entity in the knowledge graph corresponding to that state.  These two requirements allow Auger\xspace to connect a user's data to the knowledge graph and to retrieve relevant data for the user to forage from.


In the experiments in this paper, we connect Auger\xspace to Wikidata.  It meets both requirements listed above: it responds to the SPARQL query language so gathering neighborhood data about entities is simple, and it has a service \verb wbsearchentities \hspace{1mm}to map strings to entities on the graph.  But it also has some extra features that we take advantage of.  Wikidata contains a broad set of information making it easy to find related attributes for many different kinds of datasets. It also has additional metadata about the data in its graph including data-type information and human readable descriptions and labels for each node and edge in the graph.  Lastly, due to its popularity, there is a large amount of documentation and guidance on constructing complex queries.



\section{Tasks and Goals}



Auger\xspace is developed as part of the DARPA Data-Driven Discovery \DIFdelbegin \DIFdel{and Modeling }\DIFdelend \DIFaddbegin \DIFadd{of Models }\DIFaddend (D3M) program whose goal is to develop software infrastructure and algorithms to make automated machine learning accessible to general data scientists. Inspired by the observation that both the use of exploratory visualization and the use of advanced machine learning are fundamentally limited by the input data, Auger was developed to help a data scientist craft better predictive models by foraging for additional \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to add to their dataset.

We conducted interview sessions with four teams within the DARPA D3M program developing applications for data exploration and predictive modeling. The goal of the interview sessions was to better understand how a tool like Auger\xspace can help the user to perform data augmentation for the purpose of improving machine learning model performance and accuracy.
Each of the interviewed teams was shown an early implementation of Auger\xspace that was able to search for related attributes and return a list of them, but without any visualizations. The participants were then asked about what additional \DIFaddbegin \DIFadd{system }\DIFaddend features and interactions would facilitate the discovery of the types of data that they were interested in for their applications.  
\begin{figure*}[thb]
\centering
\includegraphics[width=0.92\textwidth]{figures/full_system_light_acled_labels.png}
\caption{The \DIFdelbeginFL \DIFdelFL{full system }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{user interface of }Auger\xspace\DIFaddFL{, }\DIFaddendFL showing an analysis session on the ACLED dataset.  \textbf{A} shows the initial list of attributes in the table as well as augmented attributes, like the maximum inflation rate recorded for each country.  \DIFdelbeginFL \textbf{\DIFdelFL{B}}
\DIFdelFL{shows the }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{The }\DIFaddendFL list of attributes related to the \DIFdelbeginFL \DIFdelFL{Country }\DIFdelendFL \DIFaddbeginFL \textit{\DIFaddFL{Country}} \DIFaddendFL column \DIFdelbeginFL \DIFdelFL{as discovered from Wikidata}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{is shown in  }\textbf{\DIFaddFL{B}}\DIFaddendFL .  \DIFdelbeginFL \DIFdelFL{The }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Each related }\DIFaddendFL attribute \DIFdelbeginFL \DIFdelFL{of the returned attribute }\DIFdelendFL is \DIFdelbeginFL \DIFdelFL{encoded in }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{adorned with }\DIFaddendFL a \DIFdelbeginFL \DIFdelFL{symbol, a color, and a label on the left side of each row.  The }\DIFdelendFL donut chart \DIFdelbeginFL \DIFdelFL{in each row shows }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{showing }\DIFaddendFL the estimated \DIFdelbeginFL \DIFdelFL{amount of rows of the dataset that will have data available in the connected knowledge graph}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{join quality}\DIFaddendFL .   \DIFdelbeginFL \DIFdelFL{When a row in either of the two columns is clicked, additional metadata }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Additional details }\DIFaddendFL about \DIFdelbeginFL \DIFdelFL{that }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{any }\DIFaddendFL attribute is \DIFdelbeginFL \DIFdelFL{shown }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{available on demand }\DIFaddendFL in a popup, as seen in \textbf{C}.  As attributes are added to the dataset \DIFaddbeginFL \DIFaddFL{during the analysis session}\DIFaddendFL , a preview of the table is updated \DIFdelbeginFL \DIFdelFL{in the raw data table }\DIFdelendFL as seen in \textbf{D}.}

    \vspace{-3mm}

    \label{fig:system_overview}
\end{figure*}


\DIFaddbegin 

\subsection{\DIFadd{Task Analysis}}

\DIFaddend We distilled a list of tasks from the interviews that would enable a user to meaningfully augment their dataset with additional attributes.

\begin{enumerate}[label=\textbf{T\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{View a list of joinable attributes for any existing attribute in the dataset.}  Revealing the list of potential attributes that can be joined with a particular column in the dataset helps the user determine what external data is available for augmentation.  It is a key aspect of the exploration process in information foraging because it may reveal data that the user wasn't aware of.
   
    \item \textbf{Analyze the properties of possibly related attributes \textit{before} the join occurs:}
    Before joining a new attribute into a dataset, users will need to gather information about the new attribute to gauge the new attribute's impact on the quality of the dataset.
    For this reason, users should be provided with as much information as possible about the potentially joinable attributes before the join.
    This might include metadata, examples of the attribute, and any available information about the general distribution of the new data.  In addition, it is important to communicate whether there will be any missing values in the joined attribute.  
   

    \item \textbf{Specify aggregations:}
    The relationships between entities in the user's dataset and entities in the external data source are varied - they could be one-to-one, one-to-many, or many-to-many.
    In order to fit information encoded as plural relationships into a single row of data, aggregations must be specified.
    For example, a user might want to add the populations of a list of states into the their data. However, because there could be multiple measures of the state's population over the years, the user would need to perform an aggregation function over these results, such as \textsc{minimum}, \textsc{maximum}, or \textsc{average} depending on their analysis need.

    \item \textbf{Connect through to additional data:}
    Attributes relevant to the user's analysis goals may not always be directly encoded as a property of an entity that exists on the initial dataset.
    For example, a user may want to augment a dataset of countries with the population of each country's capital.
    This information may not be directly accessible as a property of each country in the data repository.
    Instead, population size is linked to the entity that represents the capital, which in turn is linked to the country.
    To support these cases, users should be supported to join \textit{through} intermediate attributes giving them fine-tuned control over how joins pass through multiple relationships.
   
   
   
   
   
\end{enumerate}

\DIFaddbegin \subsection{\DIFadd{Design Goals}}

\DIFadd{Based on this task analysis, we iterated through several sketches to come up with a set of design goals.  In the subsequent section, we describe a system with visualizations and interactions that address these goals.  However, this list of design goals should prove to have a broader impact than }Auger\xspace \DIFadd{alone - they can inform how data augmentation can be integrated into other visual analytics tools.
}

\begin{enumerate}[label=\textbf{G\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{\DIFadd{Attributes as first-class citizen.}}  \DIFadd{Because external data is found based on attributes in the original data (}\textbf{\DIFadd{T1}}\DIFadd{), the list of attributes is the most important data to encode.  In early sketches, we used a data table to show the user what the data looked like, and allowed users to click on column headers to view new attributes and view joined data in enriched cells, similar to the }\textit{\DIFadd{ETable}} \DIFadd{from Kahng et al~\mbox
\cite{kahng_2016}}\hspace{0pt
.  We ultimately decided that a table encoding dedicated too much space to the table formatting and data found in the table cells.  Instead, we settled on a primary view that listed all attributes of the dataset.  By viewing the list of attributes, users are able to build a story about what attributes might still be needed for their analysis.  This list can update as new attributes are foraged.  Additional details about each attribute, such as summary statistics and examples, can be provided on demand.
   
    }\item \textbf{\DIFadd{Visual exploration of joinable data:}} \DIFadd{In order to discover interesting joinable data (}\textbf{\DIFadd{T2}}\DIFadd{) and specify aggregations on that data to create new attributes (}\textbf{\DIFadd{T3}}\DIFadd{), users should be able to explore the set of joinable data.  Visual cues can help in this exploration, by encoding meaningful information about external data like the join quality or the data distribution.  As the dataset evolves, users should be given previews of the data they have added to enable iterations of exploration.
    }\item \textbf{\DIFadd{Provide visual examples for complex queries.}} \DIFadd{In practice, building complex queries that stepped through intermediate information on the knowledge graph becomes very complicated because it requires multiple aggregations over the joined data.  For these types of complex queries, visualizing the knowledge graph about one example in the data can help a user understand how they are combining and shaping the data.
}\end{enumerate}

\DIFaddend \section{Auger\xspace: A Visual Analytics System}

In this section, we describe the function and design of Auger\xspace.  \DIFdelbegin \DIFdel{We start by describing the }\DIFdelend \DIFaddbegin \DIFadd{The general workflow of }Auger\xspace \DIFadd{starts when a user uploads a tabular dataset.  The user can then search for related attributes of any column in their dataset containing entities that can be found in Wikidata.  They can augment their dataset with these related attributes by specifying aggregations.  After several iterations, if the user is satisfied with the augmented dataset, they can export their data for further analysis in other visual analytics tools.  
}

\DIFadd{In this section, we first describe the }\DIFaddend design of each component of the interface and their interactions.  Then, we \DIFdelbegin \DIFdel{give details on the implementation of the back-end connection to the Wikidata }\DIFdelend \DIFaddbegin \DIFadd{explain how these user interactions are translated into queries that can be executed over a }\DIFaddend knowledge graph.






\subsection{User Interface}

\DIFdelbegin \DIFdel{Here, we describe the general look and feel of 
\DIFdel{and go into depth about how its design enables the tasks listed above. As shown in }\DIFdelend
Figure~\ref{fig:system_overview}, Auger\xspace has the following main interface components:

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/through_join_small_with_labels.png}
    \vspace{-4mm}
    \caption{When a through join is specified, a dialog window pops up to allow the user to specify aggregations at each level of the join.  To help them specify that query to the system, Auger\xspace shows them an example of the neighborhood in the knowledge graph that is queried for a single row of the table.  In this image, the user would like to add the lowest life expectancy of any neighboring countries because they believe this will help their analysis of conflict.  And for each country, several values are available for life expectancy, so the user selects the ``max'' operator.  All six neighbors for Iraq, but only three are shown for simplicity of the illustration.  }
    \vspace{-3mm}

    \label{fig:through_join}
\end{figure}







 \vspace{3pt} \noindent \textbf{Column View:} This view shows the attributes and their data types (\DIFdelbegin \DIFdel{represented using colored icons) that are present in the loaded data (}\DIFdelend \DIFaddbegin \textbf{\DIFadd{G1}}\DIFadd{, }\DIFaddend see Figure~\ref{fig:system_overview}-A). \DIFdelbegin \DIFdel{As the user discovers new attributes, they can remove any particular column if they find it is not helpful to their analysis by a simple toggle button.  Users can click on any row of this view to see }\DIFdelend \DIFaddbegin \DIFadd{Users can view details on demand about each attribute, including }\DIFaddend the data distribution of that attribute shown as a histogram chart, \DIFdelbegin \DIFdel{along with any available meta-data describing that attribute (}\DIFdelend \DIFaddbegin \DIFadd{several example values, the type of unit (i.e. kilogram or mile, if applicable) (}\DIFaddend see Figure~\ref{fig:system_overview}-C).

 \vspace{3pt} \noindent \textbf{Related Attributes:} A user can \DIFdelbegin \DIFdel{toggle a search for a collection of }\DIFdelend \DIFaddbegin \DIFadd{search for }\DIFaddend related attributes of any \DIFdelbegin \DIFdel{string-like }\DIFdelend \DIFaddbegin \DIFadd{string }\DIFaddend attribute in the Column View\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T1}
\DIFdel{)}\DIFdelend .  Auger\xspace shows the \DIFdelbegin \DIFdel{retrieved attributes from the connected knowledge graph in a second column }\DIFdelend \DIFaddbegin \DIFadd{related attributes discovered from the knowledge graph }\DIFaddend next to the list of attributes of the current table, as seen in Figure~\ref{fig:system_overview}-B. 
\DIFdelbegin \DIFdel{Each row in this list represents a potential attribute that can be joined to the selected column.  Similar to attributes in the column view, metadata and distributions of each attribute can be viewed in a tooltip, as seen in Figure~\ref{fig:system_overview}-C.  The numbers surrounded by colored circles, also known as donut charts, represent the estimated percent of rows of the dataset that have joinable data for that attributein the knowledge graph.
}\DIFdelend
 \DIFaddbegin \DIFadd{An estimation of the join quality of each related attribute is encoded visually with a donut chart, and details on demand are also provided for each attribute, including distribution of values.
 }\DIFaddend This information can help the user identify whether a related attribute has the potential to help their analysis \textit{before} joining it to their data (\DIFdelbegin \DIFdel{Task }\DIFdelend \textbf{\DIFdelbegin \DIFdel{T2}\DIFdelend \DIFaddbegin \DIFadd{G2}\DIFaddend }).  Both the estimated probability and the estimated attribute distribution are calculated from random sample of possible joinable data, since calculating the true values would require completing the full join for every row in the dataset.  For more details on implementation techniques, see section~\ref{sec:technique}.

 \vspace{3pt} \noindent \textbf{Adding Attributes:} To add any attribute, users click the plus symbol, and Auger\xspace will show a drop-down menu revealing various \DIFdelbegin \DIFdel{join-operations users can perform to join this attribute to the data}\DIFdelend \DIFaddbegin \DIFadd{options for join operations}\DIFaddend . The available join operations correspond to the data type of the relationship.  If the relationship is one-to-one, i.e. a country has a single head of government, then a single value will be retrieved for each row of the dataset.  If the relationship is one-to-many or many-to-many, an aggregation must be specified.  For example, if the attribute is a collection of numbers, such as set of populations recorded for a country, the user can select numerical aggregations such as \DIFaddbegin \textit{\DIFadd{count}}\DIFadd{, }\DIFaddend \textit{mean}, \textit{max}, \textit{min}, \textit{sum}, or \textit{variance}\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T3}
\DIFdel{).  }\DIFdelend \DIFaddbegin \DIFadd{.  If the attribute is a datetime, the user can select from }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{max}}\DIFadd{, or }\textit{\DIFadd{min}}\DIFadd{.  If the attribute is a string, the user can select either }\textit{\DIFadd{count}} \DIFadd{or }\textit{\DIFadd{through}}\DIFadd{, a type of placeholder operation that let's the user step through this attribute to form complex multi-hop joins (more details are provided in the next bullet).  In addition to these join operations, for any type of data, the user can select to randomly sample from the collection.  This operation is useful if an attribute is generally expected to have only one value per entity, but due to data quality issues, some rows might have it recorded twice, like date of birth.  
 }\DIFaddend 

 It can be very costly to resolve all of the queries needed to join an additional attribute to an entire dataset; gathering the data from the knowledge graph requires the retrieval of many different parts of the knowledge graph \DIFdelbegin \DIFdel{rather than }\DIFdelend \DIFaddbegin \DIFadd{in contrast to }\DIFaddend the retrieval of a single column from a relational database.  Auger\xspace mitigates much of that time cost by only joining attributes for the top 10 rows of the dataset.  Users are still able to glance at the dataset preview, as seen in Figure~\ref{fig:system_overview}-D, and get an idea of the data that is being joined to the dataset \DIFaddbegin \DIFadd{(}\textbf{\DIFadd{G2}}\DIFadd{)}\DIFaddend .  This lets the user explore the available attributes rapidly.  When the exploration phase is completed, the full join can be executed.  

 \vspace{3pt} \noindent \textbf{Through Joins:} In some cases, the user may want to join to data through an intermediate attribute.  If the intermediate attribute has a one-to-one relationship with the rows of the original dataset, i.e. a country has one head of government, then the user can simply join the intermediate attribute first, and then use that as their starting point to seek more related attributes.  If the intermediate attribute is a one-to-many relationship, then the user must specify multiple levels of aggregation.  We call this type of aggregation a \textit{``multi-hop aggregation,''} since it requires aggregating data across multiple hops on the knowledge graph.  

 As an example, suppose the user wants to determine if a country has a lower life expectancy than its neighbors.  Consider how that would be calculated for a single country, Iraq.  To gather the required data from the knowledge graph, the user must first join to all bordering countries, then connect through them to reach the life expectancies of those countries.  Once all data is selected, aggregations must be specified to produce the desired value.  In Wikidata, countries have multiple life expectancies recorded, so the user has to specify that they want the maximum life expectancy recorded.  That value is then calculated for each bordering country (i.e. Iran, Turkey, Jordan, etc.).  Then, they have to specify that they want the minimum among all bordering countries.  Expressing this type of query is complicated enough to explain for a single value, let alone for an entire column.  
 \DIFaddbegin 

 \DIFaddend In Auger\xspace, the user is shown a simplified illustration of the topology of the knowledge graph to assist them in understanding the aggregations that the user must choose, as seen in Figure~\ref{fig:through_join}.  \DIFaddbegin \DIFadd{For a given row of the dataset, up to three values of the intermediate attribute and the target attribute are sampled from the knowledge graph.  The system then displays these values on a graph, representing a small slice of the knowledge graph.  Users can then dynamically set the aggregations (i.e. }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{mean}}\DIFadd{) that occur at each level of the join graph, and see the resulting value that would get joined for that row.  }\DIFaddend By viewing the values that will get \textit{passed through} the knowledge graph to ultimately calculate the attribute for one row of their data, users get reinforcement that the complex query they are building \DIFdelbegin \DIFdel{results in reasonable data.
 }\DIFdelend \DIFaddbegin \DIFadd{in }Auger\xspace \DIFadd{matches the query they are building in their head (}\textbf{\DIFadd{G3}}\DIFadd{).
 }\DIFaddend 

\DIFdelbegin
\textbf{\DIFdel{Showing the augmented data: }}
\DIFdel{When additional attributes are successfully added to the data, the column view shows a card on the right showing the list of attributes that are added to the data. Added columns are also shown as a nested list in the first card below the parent attribute through which it was retrieved from the knowledge graph, as seen in Figure~\ref{fig:system_overview}-A. If the added column is a string, it can then be used as a starting point to search for more related attributes.  In that manner, the user can reach further and further away from the initial attribute set by iteratively adding attributes As attributes are added, the table view updates with the new list of columns and shows the user example values for the top 10 rows of the dataset (see Figure \ref{fig:system_overview}-D).

}\DIFdelend




 









\subsection{Backend Implementation}

\label{sec:technique}

Here, we describe how our visual analytics system augments a dataset by translating the tasks supported by the user interface described above into valid queries over a knowledge graph.  Auger\xspace gathers data from the connected knowledge graph in two different subroutines.  In all examples in the paper, Auger\xspace connects to Wikidata using the SPARQL query language.\footnote{Example queries generated by the system for both subroutines are available in the supplemental material.}



\vspace{3pt} \noindent \textbf{Finding Related Attributes:} This subroutine receives a column of data as input \DIFdelbegin \DIFdel{(i.   e. the Country columnand all it's data, }\DIFdelend \DIFaddbegin \DIFadd{and returns a list of attributes along with assorted metadata.   Consider a dataset with a Country column, i.e. }\DIFaddend ``Germany,'' ``France,'' ``Austria,'' etc.\DIFdelbegin \DIFdel{), and returns a list of attributes along with assorted metadata.  
In }\DIFdelend \DIFaddbegin \DIFadd{.
Unlike a column in a SQL table, where entities and their relations might be expected to adhere to a schema, a list of countries in }\DIFaddend a knowledge graph \DIFdelbegin \DIFdel{, although these are all entities about a country, they }\DIFdelend might not share the same set of attributes\DIFdelbegin \DIFdel{(i. e. they might have different numbers of edges connected to the entities, and those edges might describe different types of relationships).  
}\DIFdelend \DIFaddbegin \DIFadd{. For example, in Wikidata, some Western countries, such as Germany, can have more than a thousand attributes, while other countries have only a fraction of that.  
}\DIFaddend In order to provide a user with a consistent set of attributes for these entities, we first need to find their commonalities.
As such, the primary goal of this subroutine is to return the list of attributes that are available on as many of the entities as possible.  
\DIFaddbegin 

\DIFaddend Checking the related attributes of each entity to gather this list can take prohibitively long, especially if the input data contains thousands to millions of rows.  \DIFdelbegin

\DIFdelend As an optimization step to make this subroutine support a user's interactive analysis, we employ a sampling-based technique to reduce computation time. With this optimization, some subset of the dataset is randomly sampled; \DIFdelbegin \DIFdel{empirically }\DIFdelend \DIFaddbegin \DIFadd{in our experiments }\DIFaddend we have found that 20-30 rows are sufficient \DIFaddbegin \DIFadd{to differentiate high-quality attributes from poor-quality ones}\DIFaddend . Each sampled row is mapped to an entity on the knowledge graph, and a list of related attributes for that row is retrieved.  Then, the lists for all the rows are compared, and the 50 attributes that appear on the most lists are returned.  \DIFdelbegin \DIFdel{The percent of lists that the attribute appears on is passed on as metadata, and is visually encoded in 
\DIFdel{as a colored donut chart, seen in each attribute row in Figure~\ref{fig:system_overview}-B.
    

\DIFdel{Various metadata is returned, including three examples for each attribute.  To get the data for the histograms, seen in Fig~\ref{fig:system_overview}-C, an additional query is run for each of }\DIFdelend \DIFaddbegin \DIFadd{In this query, we also gather all details of }\DIFaddend the \DIFdelbegin \DIFdel{50 related attributes to get 20-30 examples through which to get a rough estimate of the distribution of the data.This subroutine runs queries in parallel and is only limited in speed by the concurrency limits on the knowledge graph API. }\DIFdelend \DIFaddbegin \DIFadd{attributes that are displayed on demand, such as the data distribution of that attribute shown as a histogram chart, several example values, and the type of unit (i.e. kilogram or mile, if applicable).  }\DIFaddend For the types of exploration done in the user study and in examples given in this paper, retrieving the list of related attributes from Wikidata typically takes 2-5 seconds.

    \DIFaddbegin 

\DIFaddend \begin{figure*}[ht]
    \centering
    \includegraphics[width=\textwidth]{figures/use_case_1_tableau.png}
        \vspace{-3mm}

    \caption{Three steps of analysis of an armed conflicts dataset.  Visualizations show the number of records of each event type by (a) country, (b) basic form of government, and (c) government type.  Iniitally, only country is available.  Auger\xspace is used to gather basic form of government first, then, after some analysis by the user, it is used to gather the government type to better understand patterns in the event types.  Visualizations generated with Tableau. }
        \vspace{-3mm}
    \label{fig:tableau_use_case}
\end{figure*}

    
\vspace{3pt} \noindent \textbf{Materializing Joins:} This subroutine takes in join instructions and returns an augmented dataset with an additional column.  The join instructions specify the path taken through the knowledge graph, along with any aggregation functions, such as ``count'' or ``max.''  Auger\xspace builds a SPARQL query to crawl the knowledge graph to retrieve the desired data (see Fig~\ref{fig:teaser}).  The time to retrieve data scales linearly with the number of rows and is limited by the concurrency limits on the knowledge graph API. While the user is interactively exploring attributes, Auger\xspace will only materialize the join for the top 10 rows of the data table to provide the user with a preview of the table in real time (see Fig~\ref{fig:system_overview}-D).  When the foraging is done, all joins are materialized for the entire dataset (rather than just a sample of rows) in the order in which they were constructed by the user, which takes around 10-15 seconds on the datasets explored in the user study in section~\ref{sec:user_study}.




Auger\xspace is implemented with a VueJS frontend web application and a NodeJS backend to handle all asynchronous calls.  Queries are sent concurrently whenever possible.  Source code is withheld to maintain anonymity and will be released in the final publication.







\label{sec:system}


\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 1: Conflict Data}



To demonstrate the value of column augmentation  for insight generation, we present a \DIFdelbegin \DIFdel{use case }\DIFdelend \DIFaddbegin \DIFadd{usage scenario }\DIFaddend in which Auger\xspace is integrated into a typical workflow for the popular visual analytics tool, Tableau.   Suppose a political scientist wants to study the factors that lead to armed conflict.  They download the Armed Conflict Location \& Event Data Project (ACLED) dataset for April 2018, which contains hundreds of records of armed conflicts across many different countries and times\cite{raleigh2010introducing}.  Initially, they load the data into Tableau to analyze relationships between countries and event types.  They produce a heatmap, seen in Figure~\ref{fig:tableau_use_case}(a), to try to detect patterns, but find it difficult to visually group countries without more data.

They load their dataset into Auger\xspace to look for additional attributes that may be relevant to event types.  When the system is first loaded, the user sees a list of the attributes in the uploaded table in Figure~\ref{fig:system_overview}-A.  They also can see the first five rows of the data below in Figure~\ref{fig:system_overview}-D.  

The political scientist believes the type of government could be related to the event type, so they click the \textit{related attributes} button next to ``Country'' in Figure~\ref{fig:system_overview}-A.  The system returns a list of related attributes found by scraping Wikidata, as seen in Figure~\ref{fig:system_overview}-B (\textbf{T1}).  The user scans through the list of related attributes, looking for information about the type of government.  They see that Wikidata holds the \textit{basic form of government} for each country.  The tooltip for that attribute, seen in Figure~\ref{fig:system_overview}-C, provides more information to affirm that this holds the right data by showing the user examples of the attribute, like ``unitary state,'' ``federal republic,'' and ``absolute monarchy'' (\textbf{T2}).  The user adds this attribute to the dataset.  While looking for the form of government, the user also notices additional interesting \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend for analysis and joins them for future analysis, including the max \textit{inflation rate}, minimum \textit{life expectancy}, mean \textit{nominal GDP per capita}, and the mean \textit{Human Development Index} (\textbf{T3}).

The user loads this next iteration of the dataset into Tableau, and generates the same visualization but with the basic form of government replacing the country name, seen in Figure~\ref{fig:tableau_use_case}(b).  While some patterns begin to emerge, the user notes that the basic form of government is too fine-grained for the type of analysis they want to do, since most categories only have one or two countries in them.  They return to Auger\xspace to see if there is any higher-level categorization available for that \DIFdelbegin \DIFdel{column by searching for attributes off of that }\DIFdelend attribute (\textbf{T4}).  They find that Wikidata holds a \textit{subclass of} attribute for basic forms of government with examples including ``democracy,'' ``monarchy,'' ``republic,'' and add that attribute using a through join before returning to Tableau to generate the final visualization, seen in Figure~\ref{fig:tableau_use_case}(c).  Now, they can see a pattern in that conflicts concentrate on republics in the region, with democracies and monarchies having interesting patterns in their types of conflicts as well.  This \DIFdelbegin \DIFdel{use case }\DIFdelend \DIFaddbegin \DIFadd{usage scenario }\DIFaddend demonstrates how the integration of column augmentation into a traditional insight generation pipeline can unlock new analyses in-situ.

\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 2: Modeling Unemployment}

\label{sec:use_cases_ml}
\begin{figure*}[ht]
    \centering
    \includegraphics[width=\textwidth]{figures/usecaseML_screenshot1_cropped.png}
    \caption{A modified version of Auger\xspace used for both \DIFdelbeginFL \DIFdelFL{Use Case }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Usage Scenario }\DIFaddendFL 2 and Task 2 of the user study. Users can build predictive models as they augment their dataset to help them discover attributes that help predict a target variable.  This figure portrays the construction of regression models to predict the rate of unemployment in counties of the United States.  The history of trained models is shown at the bottom of the column view.   By mousing over a model, the user can see the five highest feature importance scores for that model, and the list of attributes used to train that model are highlighted in black.}
    \label{fig:ml_use_case}
\end{figure*}

In our second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate the value of Auger for a separate type of visual analytics task - the generation of a predictive model.  Many visual analytics systems have been constructed to enable domain experts to interact with and steer the generation of machine learning models on their data.  Augmenting a dataset with new \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend is one way in which a user can imbue domain knowledge into the modeling process.  They may know that certain attributes can help the prediction while others may mislead.  While it may seem unusual to augment a training set with additional attributes, the additional attributes found on the knowledge graph can likewise be added to any data that the resulting model is asked to predict on at test time when the model is deployed.

We modify Auger\xspace to be able to train a predictive model at any point in the data augmentation process (see Figure~\ref{fig:ml_use_case}).  When the user instructs Auger\xspace to build a model, Auger\xspace splits the data into a training and test set in the ratio of 0.8:0.2 and trains a random forest regressor.  For each model, Auger\xspace reports the \textit{R Squared} score on the training and test sets, as well as the \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend importance scores of the five most important \DIFdelbegin \DIFdel{features}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend .  

Consider Andy\xspace is a public policy analyst who seeks to build a regression model on the unemployment rate dataset\cite{unemployment2015} with $1200$ data samples from the year $2015$. Every row in the data is a county and contains a sparse set of attributes such as \textit{County}, \textit{State}, \textit{Month}, and the response variable \textit{Unemployment Rate}. 
Andy\xspace seeks to augment this data by adding meaningful \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to accurately predict the \textit{Unemployment Rate}. 
When Andy\xspace first loads the data in Auger\xspace, an initial model is trained with an \textit{R Squared} score of $0.646$ on training and $0.292$ on test set respectively. 
Andy\xspace seeks to improve the regression model's performance further by augmenting the base data using Auger\xspace's workflow and visual interface.
Andy\xspace searches for related attributes of the variable \textit{State} from the Column View. In response, Auger\xspace shows a list of related attributes that Andy\xspace may consider to add to the data. 
From these attributes they click on a set of interesting attributes such as \textit{shares border with}, \textit{head of government}, \textit{inflation rate}, etc. to see the distribution of values, and its metadata as a text view. 

From these set of attributes Andy\xspace thinks that the attributes \textit{Inflation Rate} and \textit{GDP per capita} are good attributes to predict the \textit{unemployment rate} and thus decides to add them to the data using the join operation \textit{Median}. They notice that the newly added \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend is displayed on the Table View. However looking at a few rows of the table Andy\xspace finds quite a few cells that are empty indicating that the joined data may have missing values. 
Nonetheless, Andy\xspace clicks a button to construct a new regression model using the augmented data.
Andy\xspace notices that the \textit{R Squared} score marginally improves (new score: $0.712$ and $0.301$ on training and test set respectively). They hover the mouse over the model metric card to see the list of ``Top $5$'' attributes with their weights utilised in the regression model. 
While Andy\xspace expected to see a substantial improvement in the model performance, 
they infer that the marginal improvement is probably due to missing values in the data after the join operation. \looseness=-1

Motivated to improve the model further, Andy\xspace searches for \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend that may directly help to predict the unemployment rate per county. Based on prior knowledge, Andy\xspace understands that the population of a state may be directly proportional to the unemployment rate, and they add that attribute. 
In the process of searching for other relevant \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend , Andy\xspace notices the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Maximum temperature}. Inquisitive to see if a temperature of a state is correlated with its \textit{unemployment rate} they add it to the table.
After constructing a new regression model, Andy\xspace notices that the \textit{R Squared} score improved substantially from $0.712$ to $0.855$ and from $0.301$ to $0.338$ for the training and test dataset respectively. Happy with the progress so far they decide to remove any \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend that may not be contributing to the prediction task. First they remove the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Month} (by triggering the slider on the Column View) and then triggers Auger\xspace to construct a new regression model. As expected Andy\xspace notices that the \textit{R Squared} score did not change. Next to investigate the contribution of the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Maximum temperature}, they remove it from the data. Andy\xspace notices that the \textit{R Squared} score of the new model dropped marginally meaning that the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Maximum temperature} was helpful in improving the model. They add the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend back to the data.

In a short time, Andy\xspace has constructed a regression model with substantially better R2 score than the model trained on the base data.  They have also gained insight into which attributes are relevant to their modeling problem, which may help their understanding of the resulting machine learning model.



\section{Preliminary User Study}
\label{sec:user_study}
We evaluate Auger\xspace in a preliminary user study. The purpose of the evaluation is to validate Auger\xspace both in terms of its usability and its effectiveness in helping a user improve a machine learning model through data augmentation. 
Specifically, we hypothesize that:

\begin{itemize}
    \item \textbf{H1}: Auger\xspace allows users to accurately join external data given a written description of that data. 
   
   
    \item \textbf{H2}: Auger\xspace is able to help users discover additional data that can improve the predictive quality of machine learning models trained on the dataset.
    \DIFaddbegin \shenyu{I think we may still keep the H2 as what it was but soften its validation in both quantitative and qualitative analyses.}
\DIFaddend \end{itemize}

We recruited $6$ participants (3 Female, 3 Male), between the age of $23-36$. We required each participant to have at least an elementary knowledge of machine learning and data analysis. Due to COVID-19, we were not able to conduct an in-person study. Instead  we conducted an online study using Bluejeans\footnote{https://www.bluejeans.com/}. The participants interacted with Auger\xspace on their own computer while sharing their screen.
We provided the participants with a url to our system that was hosted on our local machine that is exposed using the Ngrok remote tunneling software\footnote{https://ngrok.com/}. The study took approximately $50-60$ minutes and we compensated the participants with a $\$10$ Amazon gift card.


\subsection{Study Design}
Before the study we asked participants to fill out a background information questionnaire regarding their name, age, gender, machine learning expertise and various use cases in which they use machine learning.
We began the study by showing the participants a tutorial video of Auger\xspace, explaining the workflow, interface GUI elements, and its interaction capabilities that support various join operations. Next we asked the participants to perform three tasks, the first of which was a practice task to ensure that they were 
sufficiently knowledgeable about Auger\xspace to perform the experimental trials. We proceeded to the experimental sessions only when we observed that the participants were confident and able to use Auger\xspace on their own. In the next two tasks we asked the participants: (1) To add a set of specified \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to a given dataset. These \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend can be added to the data using various join operations supported by Auger\xspace. (2) To freely augment the data such that they can improve the performance metric of a machine learning model (metric being a R Squared Score of a regression model). 
We used the Scikit Learn machine learning library \cite{scikit_ml} to construct \textit{Random Forest Regression} models, in the same manner as described in Section~\ref{sec:use_cases_ml}.
We collected the following data from each experimental trial: (1) \textit{Task competition time}, (2) \textit{Task Accuracy}, (3) \textit{Model performance metric} e.g., R Squared Score, and (4) \textit{User ratings} collected using a post-study Likert-scale questionnaire.


\subsection{Datasets}

For the tutorial video and practice task we used a U.S. Census Bureau dataset containing data about poverty in different counties around the U.S.\cite{poverty2020}.
The dataset contains $3136$ rows, and $6$ \DIFdelbegin \DIFdel{attributes such as }\DIFdelend \DIFaddbegin \DIFadd{attributes--the dataset index, }\DIFaddend \textit{FIPS}, \textit{State}, \textit{County}, \textit{RUCCode}, and the number of residents living in the county under the poverty line. 
For the first task in our experimental session we use the IMDB Movies dataset\cite{imdb2020} containing $500$ movies and $28$ attributes such as 
\textit{Director-name}, \textit{Duration}, \textit{Movie-Title}, \textit{Movie Cast Facebook Likes}, etc. The second task in the experimental session which required users to augment data and construct regression models, used the unemployment rate dataset\cite{unemployment2015}. This dataset contained $1200$ rows where each row corresponded to a county's poverty rate measured in a given month. This dataset contained only $5$ \DIFdelbegin \DIFdel{attributes such as }\DIFdelend \DIFaddbegin \DIFadd{attributes--the dataset index, }\DIFaddend \textit{Month}, \textit{State}, \textit{County}, and the \textit{Unemployment-Rate} (dependent variable). 
\DIFaddbegin \shenyu{As we are running out of space, maybe we should remove the detailed attribute list for each dataset and provide them as part of the supplemental materials.}
\DIFaddend

\subsection{Tasks and procedure}
Participants were first asked to complete a practice task.  
During the practice task, their answers were not recorded and their performance was not included in the analysis described below.  
For the practice task, using the Poverty dataset, we asked the participants to add three \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend : (1) the area of the county, (2) the population of the state, and (3) the earliest inception date of any county that each county shared borders with. 
While the first two \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend could be retrieved using a straight-forward \textit{join by value} operation, the third \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend required the \textit{join through} operation to search for the required \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend that was one ``hop'' away in the knowledge graph. For the first experimental task using the IMDB dataset, participants were given descriptions of four attributes to add: (1) the director's country of citizenship, (2) the number of awards received by the director, (3) the number of cast members in each movie, (4) the sum of the number of awards received by all the producers of each movie.  
Note that attributes 3 and 4 required the participants to \textit{join through} several attributes.  

The final experimental task gave participants $10$ minutes 
to augment the Unemployment rate dataset with as many attributes as they'd like that they believed would aid in building an accurate regression model to predict poverty rate.  
At any point in the 10 minutes, participants could instruct Auger\xspace to build a regression model with the current dataset to give them insight into whether they were improving the predictive modeling process.  
Building a model took about 10-20 seconds, and participants would be shown the change in R Squared Score for each new model, as well as the  weights of top ``5'' \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend used in the model. 
Random Forest regression models were used because their flexibility, speed, and accuracy met the constraints of the experiment.


\subsection{Data Collection}

After all the tasks were completed, participants were asked to fill out a post-study questionnaire (using Google Forms) that included: (1) Likert-scale\DIFaddbegin \DIFadd{\mbox
\cite{likert1932technique} }\hspace{0pt
}\DIFaddend rating questions on their use of the system, (2) a NASA-TLX\cite{hart1988development} questionnaire to measure task difficulty, and (3) Open-ended descriptive questions asking users about the interface design, the workflow, and other responses related to improve the usability of the system\footnote{Pre-experiment and Post-experiment surveys are attached as supplemental material}. 
The Likert-scale questions asked the participants to rate (in a scale of $1-7$, $1$ being ``strongly disagree'') if they found the system: (1) Easy to learn, (2) Intuitive, and (3) Expressive for data augmentation. 
We used the open-ended descriptive feedback to qualitatively evaluate the usability and the interaction design of Auger\xspace for their tasks.
Open-ended prompts included (1) \textit{Describe your thoughts about Auger\xspace}, (2) \textit{Describe things you disliked about the system or the workflow. Elaborate on how you think it could have been improved}, and (3) \textit{Describe your strategy or your process to augment the data for Tasks 1 and 2}\DIFaddbegin \DIFadd{.
}\DIFaddend With the consent of the participants,
each session was \DIFaddbegin \DIFadd{both }\DIFaddend video and audio-recorded. We encouraged the participants to verbalize their thoughts following a think-aloud protocol.
\DIFaddbegin \shenyu{Same here. As we are running out of space, maybe we should remove the details of the Likert-scale questions and open-ended prompts and provide them as part of the supplemental materials.}
\DIFaddend Furthermore, to assess if data augmentation using Auger\xspace led to any change in the regression models performance\DIFaddbegin \DIFadd{, }\DIFaddend we saved: (1) model metric (i.e., R Squared Score), (2) \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend weights, and (3) predicted values from the model. We also saved user mouse-clicks to analyze the set of \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend the user explored to augment the data.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/questionnaire_responses_2.pdf}
    \vspace{-8mm}
    \caption{Participants' responses about the system (Qn1--Qn4, the higher the better) and the overall effort (Qn5--Qn9, the lower the better).}
        \vspace{-3mm}

    \label{fig:survey_ratings}
\end{figure}


\subsection{Result and Analysis}
We report three types of results: task performance, user satisfaction, and qualitative feedback.

\subsubsection{Task Performance}

To assess \textbf{H1}, we reviewed the number of attributes that participants correctly joined in Task 1.
Five out of six participants properly joined all four expected attributes.
Only one participants missed one attribute when joining the number of awards received by the director.
This indicates that in general participants were able to use Auger\xspace to accurately add new data attributes.

Next, to assess \textbf{H2}, we similarly reviewed the \textit{R Squared} score improvement that participants achieved by augmenting additional data in Task 2.
On average, participants improved the \textit{R Squared} Score by $0.048$ ($\sigma=0.014$), starting with a baseline \textit{R Squared} Score of $0.292$. 
The area and population of the county are two most frequently joined attributes across all the participants, which is reasonable as these two attributes are usually the most influential for predicting unemployment rate. 

As a result of our analysis, we \textbf{accept \DIFdelbegin \DIFdel{both }\DIFdelend H1\DIFdelbegin \DIFdel{and H2}\DIFdelend } because all participants were able to successfully use Auger\xspace to explore and identify the correct attributes for data joining in Task \DIFdelbegin \DIFdel{1, and to improve the }\DIFdelend \DIFaddbegin \DIFadd{1.
However, we }\textbf{\DIFadd{reject H2}} \DIFadd{as the study results have no clear evidence of improving the }\DIFaddend predictive quality of a regression model in Task \DIFdelbegin \DIFdel{1. 
}\DIFdelend \DIFaddbegin \DIFadd{2.
}\shenyu{I am not sure whether we should say \textbf{accept part of H2}. By using Auger\xspace, the participants do be able to discover additional useful attributes. However, the improvement of the machine learning model (for our case, predictive quality) is not obvious, which might be restricted by the chosen datasets as a limitation of our user study.}
\DIFaddend 



\subsubsection{User satisfaction}

We use the post-study questionnaire to assess user satisfaction of the data augmentation process in Auger\xspace.
\autoref{fig:survey_ratings} shows the results of the questionnaire responses.
Although the participant size is too small to infer statistical significance using quantitative analysis, we do observe that participants found Auger\xspace easy and intuitive to use based on the likert scale user ratings (\autoref{fig:survey_ratings}, \textbf{Qn1,2}).
Participants were also satisfied with the two main functionalities provided by Auger\xspace, finding relevant attributes from the data and joining additional data (\autoref{fig:survey_ratings}, \textbf{Qn3,4}).
The mean ratings of \textbf{Qn1--4} were all 6 or above.
Furthermore, in analyzing user satisfaction related to the overall process (\autoref{fig:survey_ratings}, \textbf{Qn5--9}), we found that participants were generally satisfied with the process of conducting the tasks, as the mean ratings of \textbf{Qn5--9} are all 2 or lower.


\subsubsection{Qualitative Feedback}

To assess \textbf{H1} and \textbf{H2} from a qualitative perspective, we analysed participants’ descriptive feedback collected from post-study interviews. We also observed participants' workflow in using Auger\xspace from audio and video recordings of their computer.
We report the following three main qualitative user responses from the study:

\vspace{3pt} \noindent \textbf{Intuitive workflow:}
All the participants \DIFdelbegin \DIFdel{fount }\DIFdelend \DIFaddbegin \DIFadd{found }\DIFaddend Auger\xspace's workflow intuitive to find and augment relative data attributes.
The primary justification for Auger\xspace's intuitiveness is that Auger\xspace provides a visual interface that is easy to infer and for users to search and add relevant attributes from the data.
As \textit{P6} noted, ``\textit{I liked the interface of this tool because it was easy to navigate and add/remove columns to the base data}.''
The intuitiveness helped most participants to easily perform the requested tasks, especially Task 1.
For example, \textit{P1} described his experience in doing Task 1 as ``\textit{just looked at the question and implemented it}'' and ``\textit{it was pretty straight forward}.''

\vspace{3pt} \noindent \textbf{Two major strategies in model construction:}
When conducting Task 2, we observed two common strategies used by the participants: (1) searching for relevant attributes based on existing or prior knowledge and (2) exploring all possible combinations of available attributes to find one that results in improvement in model performance.
Some participants relied on their existing domain knowledge for augmenting the data with new attributes.
This was summarized nicely by \textit{P5}, ``\textit{I was actually using my domain knowledge. Thinking which factors can improve unemployment rate predictions, and then tried to select the variables based on the choices I had. This actually helped me improve the model's performance}.''
However, a few participants who may lack prior knowledge about the data, instead explored the dataset and tried out all possible combinations of attributes to find a model that is a significant improvement over the model trained on the base data.
``\textit{I selected columns one by one to see the change of R-squared score. By doing so, I was able to filter the variables that decrease R-square. In the end, I was able to get a relatively high R-squared model}'' (\textit{P6}).
The rest of the participants adopted a combination of both strategies.
For example, \textit{P2} described his strategies as follows, ``\textit{Thought a bit about what could boost performance, then I did some trial and error (manual feed-forward selection of attributes)}.''


\vspace{3pt} \noindent \textbf{Need for more powerful system features for expert users:}
Participants with visual analytics backgrounds expected editable visualizations to represent or encode \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend distribution differently.
``\textit{It would be cool to add additional ways of visualizing attributes}'' (\textit{P2}).
What's more, participants with database backgrounds wanted more details about queries of fetching data to increase flexibility.
``\textit{Maybe consider providing the real query of fetching the dataset to the expert users to give them more clear sense and allow them to change the query}” (\textit{P1}).
Similarly, participants with more expertise in machine learning sought more control over the model building process.
``\textit{May be add hyperparameter tuning for the model building phase, such as L1, L2 norm or change learning algorithm of the model}'' (\textit{P2}).
All these suggestions are valuable for guiding our further improvement in generalizing Auger\xspace to tackle various problem domains.



\subsection{Limitations}

While our user study results \DIFdelbegin \DIFdel{support both }\DIFdelend \textbf{\DIFaddbegin \DIFadd{support }\DIFaddend H1} \DIFdelbegin \DIFdel{and }\DIFdelend \DIFaddbegin \DIFadd{but }\DIFaddend \textbf{\DIFaddbegin \DIFadd{reject }\DIFaddend H2}, the results should be read in the light of the study limitations.
The number of the participants for our study are limited, which might be a confounding point of the results.
As we conducted the study online, the uncertainty existing in the online setup, such as internet connection, might also confound the results.
To improve the performance of the system and the fluidity of user interaction, we limited the number of related attributes that participants could fetch for each parent attribute.
Although it did help participants with a smoother user experience during the study, participants also complained about the restriction as it limited their performance of tasks, especially Task 2.
\DIFaddbegin \shenyu{While Auger\xspace does help participants discover additional useful attributes in Task2, the short task time and the limited number of accessible related attributes might confounded the results.}
\DIFaddend
Despite the limitations, the user study does help us to better identify the limitations in our system design and improve the system with new features.





\section{Discussion}
\label{sec:discussion}

Through our user study and \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we have demonstrated the efficacy of information foraging using knowledge graphs within visual analytics systems.  The generality of Auger\xspace suggests that data augmentation could \DIFaddbegin \DIFadd{be }\DIFaddend added into traditional visual analytics system workflows to improve the outcome of any embedded task.  Many fruitful avenues of research arise when considering how to apply information foraging to the full space of \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend which are served by visual analytics systems.

\subsection{Mental Models of Data Augmentation}

Knowledge graphs can be difficult to reason about, especially when they are used to find data corresponding to an entire column of a dataset.  In Auger\xspace, we address this by showing the user previews of the joined dataset, as well as visualizations of the distribution of joined data and an example of the portion of the knowledge graph that is used to construct a single value (see Figure~\ref{fig:through_join}).  This approach was based on our conversations with designers of visual analytics applications for building predictive models.  A better understanding of the user's mental model of the knowledge graph and the joining process is needed to generalize this approach to other \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend .  We generally aimed to hide the complexity of the underlying knowledge graph, but it may be the case that more direct exploration of the knowledge graph is helpful for some cases.

\subsection{Design Space for Interactions with Knowledge Graphs}

In Auger\xspace, the channel between the user and the knowledge graph is limited to the two subroutines shown in section~\ref{sec:technique}, which allow the user to see lists of related attributes and then construct queries given that information.  However, there is more potential to improve the user's control over the process and address edge cases by expanding the set of interactions between user and knowledge graph.  Automated processes in Auger\xspace could be replaced by collaborations between user and system.  

For example, ambiguities in the entity resolution used to retrieve a list of related attributes can be solved by user interaction.  A list of countries could refer to the governmental entities they describe, or they could refer to the national soccer teams participating in the World Cup; there will always exist cases where disambiguating the type of \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend will require domain expertise from a user.  

The user could also benefit from more fine-grained control over the process of building aggregation queries.  Users may want to join timestamped data, which would necessitate some specification from the user of how to parse and interpret temporal columns.  Spatial data offers an additional potential, as the user might want to use geographical data in their dataset to search for the closest weather station or other geographically tagged entity.  There are many types of queries that might necessitate different user interactions than have been used in previous visual analytics systems.


\subsection{Scalability}

In order for Auger\xspace to operate at the speed necessary for exploration, it makes use of only samples of the data, rather than the entire dataset .  When gathering the list of related attributes for a column, it samples only a subset of values from that column.  Those related attributes are then shown with estimations of their distribution as well as what percent of rows have a joinable value.  While we use a sampling-based approach, there is plenty of opportunity to develop methods that provide better estimates or allow for joining to more data in less time.  
The database community is working on tools for query optimization over knowledge graphs\cite{Mohamed2020}, which will help.  But specific indices, query optimizations, and data structures should be explored by the visual analytics community, the same way that many advancements in querying tabular data have been made to allow for faster data exploration\cite{lins2013nanocubes, wang2018neuralcubes, pahins2016hashedcubes, liu2013immens, tao2019kyrix}.






\section{Conclusion}
\label{sec:conclusion}
In this work, we presented Auger\xspace, a visual analytics system for explorative information foraging using knowledge graphs.  Knowledge graphs offer a wealth of information on a broad array of topics that could be used to improve the outcome of embedded tasks of visual analytics systems.  In our \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we demonstrated that the data gathered through Auger\xspace could result in better predictive models and better insight generation on two datasets.  And in our experiment, we showed that Auger\xspace is simple to learn and use, as all six of our participants were able to effectively explore and join data on multiple datasets within a 60 minute session.  


\acknowledgments{
We thank our collaborators in DARPA's Data Driven Discovery of Models (d3m) program.  This work was supported by National Science Foundation grants IIS-1452977, 1162037, and 1841349, as well as DARPA grant FA8750-17-2-0107.
}

\bibliographystyle{abbrv-doi}


\subsection{New Intro}



Over 20 years ago, Pirolli and Card coined the term ``information foraging'' to denote the process of seeking and gathering information to apply towards a task\cite{pirolli1999information, pirolli2005sensemaking}.  While they focused on foraging for additional entities or rows of a dataset, foraging for additional data attributes or columns can unlock new analytical capabilities.  For example, if a dataset contains a list of countries, adding the population of those countries as an attribute enables per-capita analyses. 

Traditionally, crafting new attributes and augmenting a dataset with them is done prior to any visual analysis.  Incorporating this process into visual analytics workflows can benefit both the augmentation process as well as the underlying tasks of the system in several ways.  First, the need for an additional attribute may only arise based on insights generated during analysis; if data augmentation is embedded in the system, users can toggle back and forth between analysis and augmentation.  Second, the process of discovering data and crafting a new attribute is an analytical task in its own right, and typically consists of complex querying that can be abstracted away by a visual interface.  Lastly, user-driven curation of the attributes of the dataset can serve as an additional medium of communication between user and system: the user can communicate domain knowledge by adding new attributes, and can likewise learn from the attributes that are discovered by the system.

However, there are many complications in integrating data augmentation into visual analytics systems.  Finding external data and matching it to the entites in the user's dataset is nontrivial.  Once connected to external data, it can be difficult to determine which data is relevant or useful to the user's analysis.  Some attributes may require the aggregation of multiple pieces of external data and can require joining through several intermediate entities.  These hurdles should be abstracted away so that the user is not required to be a database expert.  

Large scale information repositories provide the potential for interactive data augmentation because they put the potential data at the user's fingertips. However, in order to use such repositories a number of challenges must be addressed. If the repository is in the form of a data lake (a large collection of tables, e.g. WikiTables\cite{bhagavatula2013methodswikitable}), it can be difficult to determine which tables are relevant and joinable to the user's dataset. Tabular data can often be fragmented as well: for example, if a user is foraging for data on water usage across the United States, each municipality might be responsible for publishing its own data, and the formats may not align, resulting in sparse, error-prone joins.  Joining together multiple tables can also result in ambiguities over the type of join (left, right, outer, inner) that are difficult to resolve, and limit the types of attributes that can be constructed without additional data munging.  In addition, plenty of external data may be irrelevant or even harmful to the analysis tasks at hand.  
While there have been great strides in addressing some of the technical issues in using data lakes (see section~\ref{sec:related_work:data_augmentation}), little work has been done to allow a user to explore potential attributes and construct new ones.  Knowledge graphs offer an alternative to data lakes because they simplify entity matching and joining ambiguities due to their graph format.  They bring different challenges, however, in scalability and complexity, and there is still a dearth of tools available to facilitate foraging over knowledge graphs.

In this paper, we present Auger\xspace, a visual analytics system for Columnar data Augmentation through Visual Analytics.  Carefully crafted attributes are synthesized by running queries over knowledge graphs, as illustrated in Figure~\ref{fig:teaser}, before being added to a dataset as additional columns for downstream analysis.  First, each row of a dataset is mapped to an entity in a knowledge graph. Then, to help the user identify potential information of interest, Auger\xspace explores the local graph neighborhood about entities in the dataset to determine commonly held attributes.  Using visualizations of data quality, distribution, and the local topology of the knowledge graph, Auger\xspace guides the user through crafting additional attributes of data without any programming or explicit querying.  Users can express complicated operations over the knowledge graph by interacting with these visualizations. In sum, Auger\xspace abstracts away the complexity of searching, retrieving, and joining data so that the user can focus on the exploratory task of determining which attributes are relevant to their analysis.



Auger\xspace shows the promise of integrating column augmentation as a foraging process within a VA system.  The design process presented in this work in section~\ref{sec:system} can be useful for future visual analytics tools to adopt columnar data augmentation as part of their workflow.  We claim that a resulting VA system can make the exploration and discovery of additional attributes interactive, and let users construct complex attributes without programming.  Users are able to discover new attributes as well as create attributes they had in mind.  The construction of the augmented dataset can improve the outcome of downstream analytical tasks, such as insight generation,  predictive modeling, or any other number of tasks that rely on the presence of a robust dataset.  

We offer evidence of these claims by describing two usage scenarios for insight generation and predictive modeling.  In the first usage scenario, we demonstrate how a domain scientist's analysis can be enriched through cycles of in-situ column augmentation and analysis.  And in the second usage scenario, we show how a user can craft attributes that result in significantly more accurate machine learning models.  We also conduct a preliminary user study on two datasets to assess the usability of Auger\xspace.  The results confirm that users are able to both discover and craft relevant attributes easily and quickly. 






















 



In this work, we present the following contributions:
\begin{itemize}[topsep=3pt, itemsep=0pt,parsep=3pt]
    \item We present a visual analytics system, Auger\xspace, for exploratory data augmentation using knowledge graphs.  We also describe the design process of using visualization as a medium for query construction and knowledge graph exploration. 
    \item We provide usage scenarios of Auger\xspace being applied to both insight generation and predictive modeling to demonstrate the generality of our approach.    
    \item We conduct a preliminary user study to assess the usability of our system in joining semantically meaningful external data across two different tasks, offering validation of our design.
\end{itemize}



\vspace{-3mm}

\section{Background and Related Work}\label{sec:related_work}

\subsection{Knowledge Graphs}
Our system uses knowledge graphs as its knowledge representation.  Relational databases are one of the most popular formats to store data because they can be implemented and queried effectively and efficiently.
However, recently knowledge graphs have gained popularity for storing very large datasets because their entity-based model is conducive to how humans think about data.
Knowledge graphs represent data entities as nodes in a graph and the edges between those nodes are relationships between those entities.
The term \textit{knowledge graph} has been loosely used to describe collections of information.
Several different definitions have been offered in recent years~\cite{hogan2020knowledge}.
In practice the term has been used interchangeably with \textit{knowledge base} or \textit{ontology}\cite{ehrlinger2016towards}.
Structuring entities in the form of an ontology is also used to organize conceptual spaces, for example for the evaluation of visual analytics systems\cite{chen2019ontological} and visualizations that support machine learning tasks\cite{sacha2018vis4ml}.

The current popularity of knowledge graphs to store information can be traced back to 2012, when Google introduced its \emph{Knowledge Graph}.
Google's Knowledge Graph is a repository that retrieves facts about entities in search terms on their search results pages\cite{googleknowledgegraph2012, Dong2014}, and is used to power the Google Assistant\cite{googleAssistant} and Google Home voice queries\cite{googleHome}. Recent work has demonstrated that the approach can generalize to other artificial intelligence tasks such as image captioning and conversational agents\cite{huang2019knowledge,zheng2018question,hixon2015learning}.

Since 2012, Wikipedia and other Wikimedia projects have collected information into the \textit{Wikidata} knowledge graph.
It contains broad information about tens of millions of entities.
Other large public repositories exist, from DBPedia\cite{auer2007dbpedia}, a collection of data resources, to the domain-specific universal protein knowledgebase \textit{UniProt}\cite{uniprot2017uniprot} and the linguistic resource \textit{WordNet}\cite{Miller1995}.
Many of these graphs are part of the Linked Open Data Cloud\cite{linkedopendata2020}, a knowledge graph that contains information about other knowledge graphs.
All these graphs contain a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

Knowledge graphs have been become popular in many research areas such as information retrieval, Natural Language Processing (NLP), and text analysis. For example, a number of approaches (e.g.,~\cite{Dalton2014, liu2015, Rao2013, Raviv2016}) and tools (e.g.,~\cite{Ferragina2010, Gabrilovich2013}) focus on text-centric information retrieval using entity links in underlying knowledge graphs. Examples of applications include the discovery of emerging entities~\cite{Hoffart2014}, the structuring of contents into topics~\cite{Balasubramanian2010}, extracting relations~\cite{Mintz2009}, semantic search~\cite{Berant2013, Krishnan2018, Nanni2018}, or predicting the missing relations between entities~\cite{Dettmers2018, Yang2015}. Researchers in NLP and text analysis fields have also utilized knowledge graphs from different perspectives, e.g., for embedding entities and words into a continuous vector space~\cite{Wang2014}, for extracting relation facts form text~\cite{Riedel2010, Hoffmann2011}, for the generation of questions and answers pairs automatically~\cite{Reddy2017}, and for parsing and interpretation of user natural language semantically~\cite{Hakkani2014}.

In our system, Auger\xspace, users can tap knowledge repositories as a data source to augment a dataset through interactive visual support. Knowledge graphs have the advantage of providing clean, curated data which means that the need for data cleaning and matching is greatly reduced.
In addition, the entity-focused way in which knowledge graphs store information can make it easier to think about the relationship of data objects and thus helps users guide the augmentation process. 



\subsection{Data Augmentation}
\label{sec:related_work:data_augmentation}
In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new attributes (columns) for the objects.
For example, in image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}.

Approaches to add columns to a dataset typically vary in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\cite{kanter2015deep} is a common method to derive new attributes from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\cite{Song2017} for a range of models, including text processing\cite{Annervaz2018}, image classification\cite{Marino2017}, and machine translation\cite{Moussallem2019}, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\cite{Mohamed2020} helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.

Automated augmentation of datasets is of interest to the database and the data science communities.
In the database community, the goal of data augmentation can manifest in many ways. 
For example, before augmentation can take place, the first challenge is to find datasets that are suitable for joining. 
In this scenario, sometimes referred to as a ``data lake''\cite{Miller2018}, the data joining system assumes that there is a finite number of candidate datasets, and the task is to identify which of them can be joined with a user's base dataset.
Systems such as Google Goods\cite{halevy2016goods}, Infogather\cite{Yakout2012}, Octopus\cite{cafarella2009data}, Aurum\cite{fernandez2018aurum}, and work by Sarma et al.\cite{das2012finding} examine the attributes of the candidate datasets and learn the relationship between those attributes and the attributes of the input data.

One challenge of automated data augmentation that is an active area of research is entity matching. 
Entity matching refers to finding the same entity in different datasets, often using machine learning techniques. Once identified, the entity can be removed (for deduplication) or merged (for augmentation). Examples of entity matching systems include Magellan\cite{konda2016magellan}, Nadeef\cite{dallachiesa2013nadeef}, Autojoin\cite{zhu2017auto}, and work by Mudgal et al.\cite{mudgal2018deep}.

What the data joining and entity matching approaches have in common is that the systems assume little knowledge about the data. The challenge is therefore to identify the commonalities between the datasets (e.g. schema matching) to determine if the datasets or the entities within are related and therefore joinable.
In Auger\xspace, we take a different approach.
Instead of assuming little to no knowledge about the candidate data, we use a knowledge graph as the source of ``candidate data.'' 
Although knowledge graphs can be loosely structured, they contain well-defined relations between entities. 
This property of knowledge graphs allows Auger\xspace find relevant data for augmentation with certainty and ease.







While automated approaches can discover joinable attributes for a given dataset, they lack semantic knowledge of expert users.
This may result in a large number of attributes that are irrelevant to the analysis problem being added to the dataset, hindering the users in further analyzing and gaining insights from the dataset.
To make use of their semantic knowledge to choose relevant attributes, users may benefit from an exploratory process facilitated by visual analytics.  Beyond the selection of relevant attributes, users may benefit from visual explanations of complex queries.  For example, in transitive relationships that go through multiple entities, joins require multiple operations to be specified for aggregating a collection into a single value.
Auger\xspace overcomes these issues by involving a user in the process.  Attributes are only added to the dataset if the user decides that they are helpful for their analysis.  

\begin{comment}
Consider a geospatial dataset, perhaps of temperature readings, with a column representing the country in which the reading took place.  In a broad data repository such as Wikidata, over 1200 attributes are potentially joinable to countries in just a single hop on the knowledge graph.  Joining irrelevant attributes into a dataset can complicate the downstream analytical tasks because of the curse of dimensionality.  

Even if an automated approach were to only select a manageable number of attributes to join, there may be semantic constraints that render certain attributes unusable that only a user would know about.  For example, consider an analyst who is studying housing prices in the United States.  If their task is to build a model for mortgage approvals, there may be certain attributes such as race that are considered protected characteristics that legally cannot be included in the analysis.  On the other hand, they may intentionally want to include these attributes so that they can do a post-hoc fairness analysis to see if those attributes can be predicted from the rest of the dataset~\cite{chouldechova2018frontiers}.  Using automated tools can be dangerous in many analysis environments, and we posit that a human in the loop tool like Auger\xspace mitigates those dangers. It is possible that the performance of Auger\xspace could be improved by using the data discovery system of Aurum rather than Wikidata or making use of the efficient queries from RDFFrames.  In addition, in one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.  For more details on how this is handled in Auger\xspace, see section~\ref{sec:system}.
\end{comment}



\subsection{Visual Analytics Approaches}

Curating, improving, and augmenting an existing dataset has been a popular research topic in visual analytics.  This can be done to aid interpretation and sensemaking\cite{cramer2017impact} during analysis.  Other visual interactive approaches allow users to query and integrate query data from heterogeneous web or local sources.
This includes helping users extract information from textual web sources\cite{Hoeber2017}, query knowledge graphs\cite{Hoefler2014}, query large metadata-rich heterogenous data stores~\cite{Smith2006}, identify relations at attribute levels in mixed data sets~\cite{Jurgen2014}, analyse associated categories in large categorial data tables~\cite{ Alsallakh2012}, or automatically create visual representations of information stored in knowledge graphs. VAiRoma helps users extract and combine information from Wikipedia articles to create visualizations that provide insight into historical events\cite{Cho2015}.
Vispedia\cite{Chan2008} lets users interactively collect and integrate data from Wikipedia tables to create visualizations and answer analysis questions.
Atlasify enables users to relate the query concept, corresponds to a Wikipedia article, to spatial entity in the underlying reference system, e.g., countries, political figure~\cite{Hecht2012}.
All of these approaches help users access information stored in different forms, but compared to Auger\xspace do not support the process of augmenting an existing dataset with additional data.







\begin{comment}
ML support in fixing data and labels:
- Helping users with correcting recorded and user labeled data~\cite{mansoor2019delfi}
- Boxer is relevant here, too (can help to find mislabeled instances) \cite{Gleicher2020}
- Also Heimerl~et~al.~\cite{Heimerl2012} active learning is a way to effectively label datasets (adding a new column to data items). (also AL: \cite{Legg2019})
- Squares\cite{Ren2016} helps users find mislabled instances in ML (classification) models, but does not support relabeling them directly. One use case shows how mislabeled instances ca be identified. Alsallakh~\cite{Alsallakh2017} claims that as well (but no use case).
Lot of work on finding mislabeled data.  Off the top of my head, Seq2Seq vis has debugging in the title. \cite{strobelt2018s}.
- M\"uhlbacher and Piringer~\cite{Muehlbacher2013} allow users to filter low quality instances to increase model quality.
\end{comment}





















\section{Knowledge Graphs}

\label{sec:knowledge_graphs}



\begin{comment}
Knowledge graphs are a generic data format in which entities are represented as nodes on a graph and the edges between those nodes represent relationships between those entities. They are used because of their flexibility and the ease of reading and writing information about entities.  Much of the current concept of knowledge graphs can be traced back to 2012, when Google introduced its Knowledge Graph, an information repository which retrieves facts about entities in search terms on their search results pages~\cite{googleknowledgegraph2012}.  Since then, Wikipedia and other Wikimedia projects have collected information into \textit{Wikidata}, a knowledge graph with broad information about tens of millions of entities.  Other large public repositories exist ranging from the collection of datasets in \textit{DBpedia}~\cite{auer2007dbpedia} to the universal protein knowledgebase \textit{UniProt}~\cite{uniprot2017uniprot}, and in fact, many of these knowledge graphs are linked themselves in the Linked Open Data Cloud~\cite{linkedopendata2020}.  These knowledge graphs represent a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

The term \textit{knowledge graph} has been loosely used to describe collections of information.   Several different definitions have been offered in recent years, and in practice the term has been used interchangably with \textit{knowledge base} or \textit{ontology}~\cite{ehrlinger2016towards}.
\end{comment}

We define our usage of the term, knowledge graph, in this section, and describe the sufficient characteristics of a knowledge graph that we assume for our system. We assume that knowledge graphs contain entities as nodes and express attributes of those entities as edges, connecting the source entity to either another entity (to express a type of relationship) or to a literal (such as a string, number or datetime).  The number of edges connected to a node can be a proxy for the number of attributes that are held by that entity.  The flexibility of this structure allows for complex relationships to be expressed between entities, including one-to-one (i.e. a country has one head of state), one-to-many (a country is composed of multiple municipalities), and many-to-many (countries share borders with other countries non-exclusively).


Auger\xspace requires that the connected knowledge graph has a data retrieval endpoint that can respond to simple queries about entities and their neighbors.  The connected graph must also have some sort of service to map from the values in the dataset uploaded to Auger\xspace to the entities on that knowledge graph.  For example, if a dataset with U.S. states is uploaded, there must be an existing service that maps from ``New York'' or ``NY'' to the entity in the knowledge graph corresponding to that state.  These two requirements allow Auger\xspace to connect a user's data to the knowledge graph and to retrieve relevant data for the user to forage from.


In the experiments in this paper, we connect Auger\xspace to Wikidata.  It meets both requirements listed above: it responds to the SPARQL query language so gathering neighborhood data about entities is simple, and it has a service \verb wbsearchentities \hspace{1mm}to map strings to entities on the graph.  But it also has some extra features that we take advantage of.  Wikidata contains a broad set of information making it easy to find related attributes for many different kinds of datasets. It also has additional metadata about the data in its graph including data-type information and human readable descriptions and labels for each node and edge in the graph.  Lastly, due to its popularity, there is a large amount of documentation and guidance on constructing complex queries.  

In future work, we hope to allow users to connect Auger\xspace to multiple knowledge graphs.  To limit scope in this work, we focused only on Wikidata because of its high standards for data quality.  In addition, it contains general knowledge that made it an applicable information repository for a number of different datasets and usage scenarios.  This also made it easier to recruit users for the study described in section~\ref{sec:user_study}.  While different knowledge graphs bring different challenges, particularly around data sparsity (see the discussion in section~\ref{sec:sparsity}), Wikidata provided a broad enough testbed to inform the design goals and interactions supported by Auger\xspace.



\section{Tasks and Goals}



Auger\xspace is developed as part of the DARPA Data-Driven Discovery of Models (D3M) program whose goal is to develop software infrastructure and algorithms to make automated machine learning accessible to general data scientists. Inspired by the observation that both the use of exploratory visualization and the use of advanced machine learning are fundamentally limited by the input data, Auger\xspace was developed to help a data scientist craft better predictive models by foraging for additional attributes to add to their dataset.

We conducted interview sessions with four teams within the DARPA D3M program developing applications for data exploration and predictive modeling. The goal of the interview sessions was to better understand how a tool like Auger\xspace can help the user to perform data augmentation for the purpose of improving machine learning model performance and accuracy.
Each of the interviewed teams was shown an early implementation of Auger\xspace that was able to search for related attributes and return a list of them, but without any visualizations. The participants were then asked about what additional system features and interactions would facilitate the discovery of the types of data that they were interested in for their applications.  
\begin{figure*}[thb]
\centering
\includegraphics[width=0.92\textwidth]{figures/full_system_light_acled_labels.png}
\caption{The user interface of Auger\xspace, showing an analysis session on the ACLED dataset.  \textbf{A} shows the initial list of attributes in the table as well as augmented attributes, like the maximum inflation rate recorded for each country.  The list of attributes related to the \textit{Country} column is shown in  \textbf{B}.  Each related attribute is adorned with a donut chart showing the estimated join quality.   Additional details about any attribute is available on demand in a popup, as seen in \textbf{C}.  As attributes are added to the dataset during the analysis session, a preview of the table is updated as seen in \textbf{D}.}

    \vspace{-3mm}

    \label{fig:system_overview}
\end{figure*}



\subsection{Task Analysis}

We distilled a list of tasks from the interviews that would enable a user to meaningfully augment their dataset with additional attributes.

\begin{enumerate}[label=\textbf{T\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{View a list of joinable attributes for any existing attribute in the dataset.}  Revealing the list of potential attributes that can be joined with a particular column in the dataset helps the user determine what external data is available for augmentation.  It is a key aspect of the exploration process in information foraging because it may reveal data that the user wasn't aware of.
   
    \item \textbf{Analyze the properties of possibly related attributes \textit{before} the join occurs:}
    Before joining a new attribute into a dataset, users will need to gather information about the new attribute to gauge the new attribute's impact on the quality of the dataset.
    For this reason, users should be provided with as much information as possible about the potentially joinable attributes before the join.
    This might include metadata, examples of the attribute, and any available information about the general distribution of the new data.  In addition, it is important to communicate whether there will be any missing values in the joined attribute.  
   
    
    \item \textbf{Specify aggregations:}
    The relationships between entities in the user's dataset and entities in the external data source are varied - they could be one-to-one, one-to-many, or many-to-many.
    In order to fit information encoded as plural relationships into a single row of data, aggregations must be specified.
    For example, a user might want to add the populations of a list of states into the their data. However, because there could be multiple measures of the state's population over the years, the user would need to perform an aggregation function over these results, such as \textsc{minimum}, \textsc{maximum}, or \textsc{average} depending on their analysis need.
    
    \item \textbf{Connect through to additional data:}
    Attributes relevant to the user's analysis goals may not always be directly encoded as a property of an entity that exists on the initial dataset.
    For example, a user may want to augment a dataset of countries with the population of each country's capital.
    This information may not be directly accessible as a property of each country in the data repository.
    Instead, population size is linked to the entity that represents the capital, which in turn is linked to the country.
    To support these cases, users should be supported to join \textit{through} intermediate attributes giving them fine-tuned control over how joins pass through multiple relationships.
   
   
   
   
   
\end{enumerate}

\subsection{Design Goals}

Based on this task analysis, we iterated through several sketches to come up with a set of design goals.  In the subsequent section, we describe a system with visualizations and interactions that address these goals.  However, this list of design goals should prove to have a broader impact than Auger\xspace alone - they can inform how data augmentation can be integrated into other visual analytics tools.

\begin{enumerate}[label=\textbf{G\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{Attributes as first-class citizen.}  Because external data is found based on attributes in the original data (\textbf{T1}), the list of attributes is the most important data to encode.  In early sketches, we used a data table to show the user what the data looked like, and allowed users to click on column headers to view new attributes and view joined data in enriched cells, similar to the \textit{ETable} from Kahng et al~\cite{kahng_2016}.  We ultimately decided that a table encoding dedicated too much space to the table formatting and data found in the table cells.  Instead, we settled on a primary view that listed all attributes of the dataset.  By viewing the list of attributes, users are able to get a sense of what attributes might still be needed for their analysis.  This list can update as new attributes are foraged.  Additional details about each attribute, such as summary statistics and examples, can be provided on demand.
   
    \item \textbf{Visual exploration of joinable data:} In order to discover interesting joinable data (\textbf{T2}) and specify aggregations on that data to create new attributes (\textbf{T3}), users should be able to explore the set of joinable data.  Visual cues can help in this exploration, by encoding meaningful information about external data like the join quality or the data distribution.  As the dataset evolves, users should be given previews of the data they have added to enable iterations of exploration.
    \item \textbf{Provide visual examples for complex queries.} In practice, building complex queries that stepped through intermediate information on the knowledge graph becomes very complicated because it requires multiple aggregations over the joined data.  For these types of complex queries, visualizing the knowledge graph about one example in the data can help a user understand how they are combining and shaping the data.
\end{enumerate}

\section{Auger\xspace: A Visual Analytics System}

In this section, we describe the function and design of Auger\xspace.  The general workflow of Auger\xspace starts when a user uploads a tabular dataset.  The user can then search for related attributes of any column in their dataset containing entities that can be found in Wikidata.  They can augment their dataset with these related attributes by specifying aggregations.  After several iterations, if the user is satisfied with the augmented dataset, they can export their data for further analysis in other visual analytics tools.  

In this section, we first describe the design of each component of the interface and their interactions.  Then, we explain how these user interactions are translated into queries that can be executed over a knowledge graph.






\subsection{User Interface}

Figure~\ref{fig:system_overview}, Auger\xspace has the following main interface components:

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/through_join_small_with_labels.png}
    \vspace{-4mm}
    \caption{When a through join is specified, a dialog window pops up to allow the user to specify aggregations at each level of the join.  To help them specify that query to the system, Auger\xspace shows them an example of the neighborhood in the knowledge graph that is queried for a single row of the table.  In this image, the user would like to add the lowest life expectancy of any neighboring countries because they believe this will help their analysis of conflict.  And for each country, several values are available for life expectancy, so the user selects the ``max'' operator.  All six neighbors for Iraq, but only three are shown for simplicity of the illustration.  }
    \vspace{-3mm}

    \label{fig:through_join}
\end{figure}






 
 \vspace{3pt} \noindent \textbf{Column View:} This view shows the attributes and their data types (\textbf{G1}, see Figure~\ref{fig:system_overview}-A). Users can view details on demand about each attribute, including the data distribution of that attribute shown as a histogram chart, several example values, the type of unit (i.e. kilogram or mile, if applicable) (see Figure~\ref{fig:system_overview}-C).
 
 \vspace{3pt} \noindent \textbf{Related Attributes:} A user can search for related attributes of any string attribute in the Column View.  Auger\xspace shows the related attributes discovered from the knowledge graph next to the list of attributes of the current table, as seen in Figure~\ref{fig:system_overview}-B. 
 An estimation of the join quality of each related attribute is encoded visually with a donut chart, and details on demand are also provided for each attribute, including distribution of values.
 This information can help the user identify whether a related attribute has the potential to help their analysis \textit{before} joining it to their data (\textbf{G2}).  Both the estimated probability and the estimated attribute distribution are calculated from random sample of possible joinable data, since calculating the true values would require completing the full join for every row in the dataset.  For more details on implementation techniques, see section~\ref{sec:technique}.

 \vspace{3pt} \noindent \textbf{Adding Attributes:} To add any attribute, users click the plus symbol, and Auger\xspace will show a drop-down menu revealing various options for join operations. The available join operations correspond to the data type of the relationship.  If the relationship is one-to-one, i.e. a country has a single head of government, then a single value will be retrieved for each row of the dataset.  If the relationship is one-to-many or many-to-many, an aggregation must be specified.  For example, if the attribute is a collection of numbers, such as set of populations recorded for a country, the user can select numerical aggregations such as \textit{count}, \textit{mean}, \textit{max}, \textit{min}, \textit{sum}, or \textit{variance}.  If the attribute is a datetime, the user can select from \textit{count}, \textit{max}, or \textit{min}.  If the attribute is a string, the user can select either \textit{count} or \textit{through}, a type of placeholder operation that let's the user step through this attribute to form complex multi-hop joins (more details are provided below).  In addition to these join operations, for any type of data, the user can select to randomly sample from the collection.  This operation is useful if an attribute is generally expected to have only one value per entity, but due to data quality issues, some rows might have it recorded twice, like date of birth.  
 
 Extracting information for the whole dataset from the knowledge graph is too computationally demanding to be performed at interactive rates.  Gathering the data from the knowledge graph requires the retrieval of many different parts of the knowledge graph in contrast to the retrieval of a single column from a relational database.  Auger\xspace mitigates much of that time cost by only joining attributes for the top 10 rows of the dataset.  Users are still able to glance at the dataset preview, as seen in Figure~\ref{fig:system_overview}-D, and get an idea of the data that is being joined to the dataset (\textbf{G2}).  This lets the user explore the available attributes rapidly.  When the exploration phase is completed, the full join can be executed.  
 
 \vspace{3pt} \noindent \textbf{Through Joins:} In some cases, the user may want to join to data through an intermediate attribute.  If the intermediate attribute has a one-to-one relationship with the rows of the original dataset, i.e. a country has one head of government, then the user can simply join the intermediate attribute first, and then use that as their starting point to seek more related attributes.  If the intermediate attribute is a one-to-many relationship, then the user must specify multiple levels of aggregation.  We call this type of aggregation a \textit{``multi-hop aggregation,''} since it requires aggregating data across multiple hops on the knowledge graph.  
 
 As an example, suppose the user wants to determine if a country has a lower life expectancy than its neighbors.  Consider how that would be calculated for a single country, Iraq.  To gather the required data from the knowledge graph, the user must first join to all bordering countries, then connect through them to reach the life expectancies of those countries.  Once all data is selected, aggregations must be specified to produce the desired value.  In Wikidata, countries have multiple life expectancies recorded, so the user has to specify that they want the maximum life expectancy recorded.  That value is then calculated for each bordering country (i.e. Iran, Turkey, Jordan, etc.).  Then, they have to specify that they want the minimum among all bordering countries.  Expressing this type of query is complicated enough to explain for a single value, let alone for an entire column.  
 
 In Auger\xspace, the user is shown a simplified illustration of the topology of the knowledge graph to assist them in understanding the aggregations that the user must choose, as seen in Figure~\ref{fig:through_join}.  For a given row of the dataset, up to three values of the intermediate attribute and the target attribute are sampled from the knowledge graph.  The system then displays these values on a graph, representing a small slice of the knowledge graph.  Users can then dynamically set the aggregations (i.e. \textit{count}, \textit{mean}) that occur at each level of the join graph, and see the resulting value that would get joined for that row.  By viewing the values that will get \textit{passed through} the knowledge graph to ultimately calculate the attribute for one row of their data, users get reinforcement that the complex query they are building in Auger\xspace matches the query they are building in their head (\textbf{G3}).
 


 

 









\subsection{Backend Implementation}

\label{sec:technique}

Here, we describe how our visual analytics system augments a dataset by translating the tasks supported by the user interface described above into valid queries over a knowledge graph.  Auger\xspace gathers data from the connected knowledge graph in two different subroutines.  In all examples in the paper, Auger\xspace connects to Wikidata using the SPARQL query language.\footnote{Example queries generated by the system for both subroutines are available in the supplemental material.}



\vspace{3pt} \noindent \textbf{Finding Related Attributes:} This subroutine receives a column of data as input and returns a list of attributes along with assorted metadata.   Consider a dataset with a Country column, i.e. ``Germany,'' ``France,'' ``Austria,'' etc..
Unlike a column in a SQL table, where entities and their relations might be expected to adhere to a schema, a list of countries in a knowledge graph might not share the same set of attributes. For example, in Wikidata, some Western countries, such as Germany, can have more than a thousand attributes, while other countries have only a fraction of that.  
In order to provide a user with a consistent set of attributes for these entities, we first need to find their commonalities.
As such, the primary goal of this subroutine is to return the list of attributes that are available on as many of the entities as possible.  

Checking the related attributes of each entity to gather this list can take prohibitively long, especially if the input data contains thousands to millions of rows.  As an optimization step to make this subroutine support a user's interactive analysis, we employ a sampling-based technique to reduce computation time. With this optimization, some subset of the dataset is randomly sampled; in our experiments we have found that 20-30 rows are sufficient to differentiate high-quality attributes from poor-quality ones. Each sampled row is mapped to an entity on the knowledge graph, and a list of related attributes for that row is retrieved.  Then, the lists for all the rows are compared, and the 50 attributes that appear on the most lists are returned.  In this query, we also gather all details of the attributes that are displayed on demand, such as the data distribution of that attribute shown as a histogram chart, several example values, and the type of unit (i.e. kilogram or mile, if applicable).  For the types of exploration done in the user study and in examples given in this paper, retrieving the list of related attributes from Wikidata typically takes 2-5 seconds.

    


\begin{figure}[ht]
    \centering
    \includegraphics[width=0.45\textwidth]{figures/acled_gdp_fatalities.png}
        \vspace{-3mm}

    \caption{Visualizing the number of records in the ACLED dataset in April 2018 by per capita GDP, a field extracted from Wikidata.  Magnitude of marks encodes number of fatalities, while color encodes event type.    Visualization generated with Tableau. }
        \vspace{-3mm}
    \label{fig:tableau_use_case}
\end{figure}

    
\vspace{3pt} \noindent \textbf{Materializing Joins:} This subroutine takes in join instructions and returns an augmented dataset with an additional column.  The join instructions specify the path taken through the knowledge graph, along with any aggregation functions, such as ``count'' or ``max.''  Auger\xspace builds a SPARQL query to crawl the knowledge graph to retrieve the desired data (see Fig~\ref{fig:teaser}).  The time to retrieve data scales linearly with the number of rows and is limited by the concurrency limits on the knowledge graph API. While the user is interactively exploring attributes, Auger\xspace will only materialize the join for the top 10 rows of the data table to provide the user with a preview of the table in real time (see Fig~\ref{fig:system_overview}-D).  When the foraging is done, all joins are materialized for the entire dataset (rather than just a sample of rows) in the order in which they were constructed by the user, which takes around 10-15 seconds on the datasets explored in the user study in section~\ref{sec:user_study}.




Auger\xspace is implemented with a VueJS frontend web application and a NodeJS backend to handle all asynchronous calls.  Queries are sent concurrently whenever possible.  Source code and installation instructions are available at \url{https://tuftsvalt.github.io/snowcat/}.





\vspace{-0.5mm}

\label{sec:system}


\section{Usage Scenario 1: Conflict Data}



To demonstrate the value of column augmentation  for insight generation, we present a usage scenario in which Auger\xspace is integrated into a typical workflow for the popular visual analytics tool, Tableau.   Suppose a political scientist wants to study the factors that lead to armed conflict.  They download the Armed Conflict Location \& Event Data Project (ACLED) dataset for April 2018, which contains 2,279 records of armed conflicts across many different countries and dates\cite{raleigh2010introducing}.  Initially, they load the data into Tableau to analyze relationships between countries, event types, and fatalities stemming from armed conflict.  They see no clear pattern that emerges as they find it difficult to visually group countries without more data.


They load their dataset into Auger\xspace to look for additional attributes that may be relevant to event types.  When the system is first loaded, the user sees a list of the attributes in the uploaded table in Figure~\ref{fig:system_overview}-A.  They also can see the first five rows of the data below in Figure~\ref{fig:system_overview}-D.  

The political scientist believes there might be economic metrics that effect the types and severity of conflicts, so they click the \textit{related attributes} button next to ``Country'' in Figure~\ref{fig:system_overview}-A.  The system returns a list of related attributes found by scraping Wikidata, as seen in Figure~\ref{fig:system_overview}-B (\textbf{T1}).  The user scans through the list of related attributes, looking for information about the countries' economies.  They see that Wikidata holds the \textit{nominal GDP per capita} for each country.  
The user adds this attribute to the dataset.  While looking for economic data, the user also notices additional interesting attributes for analysis and joins them for future analysis, including the max \textit{inflation rate}, minimum \textit{life expectancy}, the \textit{type of government}, and the mean \textit{Human Development Index} (\textbf{T3}).

The user loads this next iteration of the dataset into Tableau, and generates various visualizations including the event types and fatality numbers with their newly-discovered attributes.  As can be seen in Figure~\ref{fig:tableau_use_case}, grouping event types by nominal GDP per capita reveals a trend that certain event types, like \textit{Remote violence} and \textit{Battle - No change of territory}, appear to have more severe fatalities in countries that have lower nominal GDP per capita.  In contrast, other event types like \textit{Battle - Government regains territory} and \textit{Battle - Non state actor overtakes territory} don't seem to be very different between high nominal GDP per capita countries and low ones.  The additional visual analysis enabled by the added attribute, nominal GDP per capita, has led the user to new hypotheses that can then be further explored.  This usage scenario demonstrates how the integration of column augmentation into a traditional insight generation pipeline can unlock new analyses in-situ.

\section{Usage Scenario 2: Modeling Poverty}

\label{sec:use_cases_ml}

In our second usage scenario, we demonstrate the value of Auger\xspace for a separate type of visual analytics task - the generation of a predictive model.  Many visual analytics systems have been constructed to enable domain experts to interact with and steer the generation of machine learning models on their data.  Augmenting a dataset with new attributes is one way in which a user can imbue domain knowledge into the modeling process.  They may know that certain attributes can help the prediction while others may mislead.  While it may seem unusual to augment a training set with additional attributes, the additional attributes found on the knowledge graph can likewise be added to any data that the resulting model is asked to predict on at test time when the model is deployed.

To demonstrate the use of CAVA in the model building process, we modify Auger\xspace to be able to train a predictive model at any point in the data augmentation process. 
Users can instruct the system to build a predictive model with the current version of the data.  Auger\xspace splits the data into a training and test set in the ratio of 0.8:0.2 and trains a random forest regressor.  For each model, Auger\xspace reports the \textit{R Squared} score on the training and test sets, as well as the feature importance scores of the five most important features.  This functionality was designed to be representative the iterative workflow of model builders, and to examine the ways Auger\xspace could help in-situ model building.

Consider Andy\xspace is a public policy analyst who seeks to build a regression model on a U.S. Census Bureau dataset containing data about poverty in different counties around the U.S.\cite{poverty2020}.
The dataset contains $3136$ rows, and $6$ attributes--the dataset index, \textit{FIPS}, \textit{State}, \textit{County}, \textit{RUCCode}, and the number of residents living in the county under the poverty line.
Andy\xspace seeks to augment this data by adding meaningful columns to accurately predict the \textit{Poverty}. 
When Andy\xspace first loads the data in Auger\xspace, an initial model is trained with an \textit{R Squared} score of $0.577$ on training and $-1.801$ on test set respectively; indicating that the model badly overfitted the training data. 
Andy\xspace seeks to improve the regression model's performance further by augmenting the base data using Auger\xspace's workflow and visual interface.
Andy\xspace searches for related attributes of the variable \textit{State} from the Column View. In response, Auger\xspace shows a list of related attributes that Andy\xspace may consider to add to the data. 
From these attributes they click on a set of interesting attributes such as \textit{shares border with}, \textit{life expectancy}, \textit{inflation rate}, etc. to see the distribution of values, and its metadata as a text view. 

From these set of attributes Andy\xspace thinks that the attributes \textit{Inflation Rate} and \textit{nominal GDP per capita} are good attributes to predict \textit{Poverty} and thus decides to add them to the data using the join operation \textit{Mean}. They notice that the newly added column is displayed on the Table View. However looking at a few rows of the table Andy\xspace finds quite a few cells that are empty indicating that the joined data may have missing values. 
Nonetheless, Andy\xspace clicks a button to construct a new regression model using the augmented data.
Andy\xspace notices that the \textit{R Squared} score marginally improves (new score: $0.586$ and $-1.810$ on training and test set respectively). They hover the mouse over the model metric card to see the list of ``Top $5$'' attributes with their weights utilised in the regression model. 
While Andy\xspace expected to see a substantial improvement in the model performance, 
they infer that the marginal improvement is probably due to missing values in the data after the join operation. \looseness=-1

Motivated to improve the model further, Andy\xspace searches for columns that may directly help to predict poverty per county. Based on prior knowledge, Andy\xspace understands that the \textit{population} of a state may be directly proportional to its poverty, and they add that attribute. 
In the process of searching for other relevant columns, Andy\xspace notices the column \textit{Maximum temperature}. Inquisitive to see if a temperature of a state is correlated with its \textit{Poverty} they add it to the table.
After constructing a new regression model, Andy\xspace notices that the \textit{R Squared} score changed from $0.586$ to $0.577$ and from $-1.810$ to $0.541$ for the training and test dataset respectively. Happy with the progress so far they decide to remove any column that may not be contributing to the prediction task. First, they remove the columns \textit{RUC Code}, and \textit{nominal GDP per capita} (by triggering the slider on the Column View) and then triggers Auger\xspace to construct a new regression model. 
As expected Andy\xspace notices that the \textit{R Squared} score did not change. 
Next Andy\xspace searches for related attributes of the column \textit{County}.
They explore the set of choices shown in the Column View.
They choose to add  the column \textit{County-population} (by median operation), and \textit{shares border with} (by count operation). Andy\xspace constructs a new regression model to see the models' train and test \textit{R Squared} score improved considerably ($0.807$, and $0.746$ respectively). Content with the improvement, they export the model and the data to continue analysis.

In a short time, Andy\xspace has constructed a regression model with substantially better R2 score than the model trained on the base data.  They have also gained insight into which attributes are relevant to their modeling problem, which may help their understanding of the resulting machine learning model.

\section{Preliminary User Study}
\label{sec:user_study}
We evaluate Auger\xspace in a preliminary user study. The purpose of the evaluation is to validate Auger\xspace both in terms of its usability and its effectiveness in helping a user improve a machine learning model through data augmentation. 
Specifically, we hypothesize that:

\begin{itemize}
    \item \textbf{H1}: Auger\xspace allows users to accurately join external data given a written description of that data. 
   
   
    \item \textbf{H2}: Auger\xspace is able to help users discover additional data that can improve the predictive quality of machine learning models trained on the dataset.
   
\end{itemize}

We recruited $6$ participants (3 Female, 3 Male), between the age of $23-36$. We required each participant to have at least an elementary knowledge of machine learning and data analysis. Due to COVID-19, we were not able to conduct an in-person study. Instead  we conducted an online study using Bluejeans\footnote{https://www.bluejeans.com/}. The participants interacted with Auger\xspace on their own computer while sharing their screen.
We provided the participants with a url to our system that was hosted on our local machine that is exposed using the Ngrok remote tunneling software\footnote{https://ngrok.com/}. The study took approximately $50-60$ minutes and we compensated the participants with a $\$10$ Amazon gift card.


\subsection{Study Design}
Before the study we asked participants to fill out a background information questionnaire regarding their name, age, gender, machine learning expertise and various use cases in which they use machine learning.
We began the study by showing the participants a tutorial video of Auger\xspace, explaining the workflow, interface GUI elements, and its interaction capabilities that support various join operations. Next we asked the participants to perform three tasks, the first of which was a practice task to ensure that they were 
sufficiently knowledgeable about Auger\xspace to perform the experimental trials. We proceeded to the experimental sessions only when we observed that the participants were confident and able to use Auger\xspace on their own. In the next two tasks we asked the participants: (1) To add a set of specified attributes to a given dataset. These attributes can be added to the data using various join operations supported by Auger\xspace. (2) To freely augment the data such that they can improve the performance metric of a machine learning model (metric being a R Squared Score of a regression model). 
We used the Scikit Learn machine learning library \cite{scikit_ml} to construct \textit{Random Forest Regression} models, in the same manner as described in Section~\ref{sec:use_cases_ml}.
We collected the following data from each experimental trial: (1) \textit{Task competition time}, (2) \textit{Task Accuracy}, (3) \textit{Model performance metric} e.g., R Squared Score, and (4) \textit{User ratings} collected using a post-study Likert-scale questionnaire.


\subsection{Datasets}

For the tutorial video and practice task we used the same U.S. Census Bureau dataset as section~\ref{sec:use_cases_ml}.
For the first task in our experimental session we use the IMDB Movies dataset\cite{imdb2020} containing $500$ movies and $28$ attributes such as 
\textit{Director-name}, \textit{Duration}, \textit{Movie-Title}, \textit{Movie Cast Facebook Likes}, etc. The second task in the experimental session which required users to augment data and construct regression models, used the unemployment rate dataset\cite{unemployment2015}. This dataset contained $1200$ rows where each row corresponded to a county's poverty rate measured in a given month. This dataset contained only $5$ attributes--the dataset index, \textit{Month}, \textit{State}, \textit{County}, and the \textit{Unemployment-Rate} (dependent variable). 

\subsection{Tasks and procedure}
Participants were first asked to complete a practice task.  
During the practice task, their answers were not recorded and their performance was not included in the analysis described below.  
For the practice task, using the Poverty dataset, we asked the participants to add three attributes: (1) the area of the county, (2) the population of the state, and (3) the earliest inception date of any county that each county shared borders with. 
While the first two attributes could be retrieved using a straight-forward \textit{join by value} operation, the third attribute required the \textit{join through} operation to search for the required attribute that was one ``hop'' away in the knowledge graph. For the first experimental task using the IMDB dataset, participants were given descriptions of four attributes to add: (1) the director's country of citizenship, (2) the number of awards received by the director, (3) the number of cast members in each movie, (4) the sum of the number of awards received by all the producers of each movie.  
Note that attributes 3 and 4 required the participants to \textit{join through} several attributes.  

The final experimental task gave participants $10$ minutes 
to augment the Unemployment rate dataset with as many attributes as they'd like that they believed would aid in building an accurate regression model to predict poverty rate.  
At any point in the 10 minutes, participants could instruct Auger\xspace to build a regression model with the current dataset to give them insight into whether they were improving the predictive modeling process.  
Building a model took about 10-20 seconds, and participants would be shown the change in R Squared Score for each new model, as well as the  weights of top ``5'' attributes used in the model. 
Random Forest regression models were used because their flexibility, speed, and accuracy met the constraints of the experiment.


\subsection{Data Collection}

After all the tasks were completed, participants were asked to fill out a post-study questionnaire (using Google Forms) that included: (1) Likert-scale\cite{likert1932technique} rating questions on their use of the system, (2) a NASA-TLX\cite{hart1988development} questionnaire to measure task difficulty, and (3) Open-ended descriptive questions asking users about the interface design, the workflow, and other responses related to improve the usability of the system\footnote{Pre-experiment and Post-experiment surveys are attached as supplemental material}. 
The Likert-scale questions asked the participants to rate (in a scale of $1-7$, $1$ being ``strongly disagree'') if they found the system: (1) Easy to learn, (2) Intuitive, and (3) Expressive for data augmentation. 
We used the open-ended descriptive feedback to qualitatively evaluate the usability and the interaction design of Auger\xspace for their tasks.
Open-ended prompts included (1) \textit{Describe your thoughts about Auger\xspace}, (2) \textit{Describe things you disliked about the system or the workflow. Elaborate on how you think it could have been improved}, and (3) \textit{Describe your strategy or your process to augment the data for Tasks 1 and 2}.
With the consent of the participants,
each session was both video and audio-recorded. We encouraged the participants to verbalize their thoughts following a think-aloud protocol.
Furthermore, to assess if data augmentation using Auger\xspace led to any change in the regression models performance, we saved: (1) model metric (i.e., R Squared Score), (2) attribute weights, and (3) predicted values from the model. We also saved user mouse-clicks to analyze the set of attributes the user explored to augment the data.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/questionnaire_responses_2.pdf}
    \vspace{-8mm}
    \caption{Participants' responses about the system (Qn1--Qn4, the higher the better) and the overall effort (Qn5--Qn9, the lower the better).}
        \vspace{-3mm}

    \label{fig:survey_ratings}
\end{figure}


\subsection{Result and Analysis}

\subsubsection{Task Performance}

To assess \textbf{H1}, we reviewed the number of attributes that participants correctly joined in Task 1.
Five out of six participants properly joined all four expected attributes.
Only one participants missed one attribute when joining the number of awards received by the director.
This indicates that in general participants were able to use Auger\xspace to accurately add new data attributes.

Next, to assess \textbf{H2}, we similarly reviewed the \textit{R Squared} score improvement that participants achieved by augmenting additional data in Task 2.
On average, participants improved the \textit{R Squared} Score by $0.048$ ($\sigma=0.014$), starting with a baseline \textit{R Squared} Score of $0.292$. 
The area and population of the county are two most frequently joined attributes across all the participants, which is reasonable as these two attributes are usually the most influential for predicting unemployment rate. 

As a result of our analysis, we \textbf{accept H1} because all participants were able to successfully use Auger\xspace to explore and identify the correct attributes for data joining in Task 1.
However, we \textbf{neither accept nor reject H2} as the study results did not contain clear evidence of users being able to improve the predictive quality of a regression model in Task 2 beyond trivial improvements.



\subsubsection{User satisfaction}

We use the post-study questionnaire to assess user satisfaction of the data augmentation process in Auger\xspace.
\autoref{fig:survey_ratings} shows the results of the questionnaire responses.
Although the participant size is too small to infer statistical significance using quantitative analysis, we do observe that participants found Auger\xspace easy and intuitive to use based on the likert scale user ratings (\autoref{fig:survey_ratings}, \textbf{Qn1,2}).
Participants were also satisfied with the two main functionalities provided by Auger\xspace, finding relevant attributes from the data and joining additional data (\autoref{fig:survey_ratings}, \textbf{Qn3,4}).
The mean ratings of \textbf{Qn1--4} were all 6 or above.
Furthermore, in analyzing user satisfaction related to the overall process (\autoref{fig:survey_ratings}, \textbf{Qn5--9}), we found that participants were generally satisfied with the process of conducting the tasks, as the mean ratings of \textbf{Qn5--9} are all 2 or lower.


\subsubsection{Qualitative Feedback}

To assess \textbf{H1} and \textbf{H2} from a qualitative perspective, we analysed participants’ descriptive feedback collected from post-study interviews. We also observed participants' workflow in using Auger\xspace from audio and video recordings of their computer.
We report the following three main qualitative user responses from the study:

\vspace{3pt} \noindent \textbf{Intuitive workflow:}
All the participants found Auger\xspace's workflow intuitive to find and augment relative data attributes.
The primary justification for Auger\xspace's intuitiveness is that Auger\xspace provides a visual interface that is easy to infer and for users to search and add relevant attributes from the data.
As \textit{P6} noted, ``\textit{I liked the interface of this tool because it was easy to navigate and add/remove columns to the base data}.''
The intuitiveness helped most participants to easily perform the requested tasks, especially Task 1.
For example, \textit{P1} described his experience in doing Task 1 as ``\textit{just looked at the question and implemented it}'' and ``\textit{it was pretty straight forward}.''

\vspace{3pt} \noindent \textbf{Two major strategies in model construction:}
When conducting Task 2, we observed two common strategies used by the participants: (1) searching for relevant attributes based on existing or prior knowledge and (2) exploring all possible combinations of available attributes to find one that results in improvement in model performance.
Some participants relied on their existing domain knowledge for augmenting the data with new attributes.
This was summarized nicely by \textit{P5}, ``\textit{I was actually using my domain knowledge. Thinking which factors can improve unemployment rate predictions, and then tried to select the variables based on the choices I had. This actually helped me improve the model's performance}.''
However, a few participants who may lack prior knowledge about the data, instead explored the dataset and tried out all possible combinations of attributes to find a model that is a significant improvement over the model trained on the base data.
``\textit{I selected columns one by one to see the change of R-squared score. By doing so, I was able to filter the variables that decrease R-square. In the end, I was able to get a relatively high R-squared model}'' (\textit{P6}).
The rest of the participants adopted a combination of both strategies.
For example, \textit{P2} described his strategies as follows, ``\textit{Thought a bit about what could boost performance, then I did some trial and error (manual feed-forward selection of attributes)}.''


\vspace{3pt} \noindent \textbf{Need for more powerful system features for expert users:}
Participants with visual analytics backgrounds expected editable visualizations to represent or encode attribute distribution differently.
``\textit{It would be cool to add additional ways of visualizing attributes}'' (\textit{P2}).
What's more, participants with database backgrounds wanted more details about queries of fetching data to increase flexibility.
``\textit{Maybe consider providing the real query of fetching the dataset to the expert users to give them more clear sense and allow them to change the query}” (\textit{P1}).
These suggestions are valuable for guiding our further improvement in generalizing Auger\xspace to tackle various problem domains.



\subsection{Limitations}

While our user study results \textbf{support H1} but \textbf{neither accept nor reject H2}, the results should be read in the light of the study limitations.
The number of the participants for our study are limited, which might be a confounding point of the results.
As we conducted the study online, the uncertainty existing in the online setup, such as internet connection, might also confound the results.
To improve the performance of the system and the fluidity of user interaction, we limited the number of related attributes that participants could fetch for each parent attribute.
Although it did help participants with a smoother user experience during the study, participants also complained about the restriction as it limited their performance of tasks, especially Task 2.
Despite the limitations, the user study does help us to better identify the limitations in our system design and improve the system with new features.





\section{Discussion}
\label{sec:discussion}

Through our user study and usage scenarios, we have demonstrated the efficacy of information foraging using knowledge graphs within visual analytics systems.  The generality of Auger\xspace suggests that data augmentation could be added into traditional visual analytics system workflows to improve the outcome of any embedded task.  Many fruitful avenues of research arise when considering how to apply information foraging to the full space of usage scenarios which are served by visual analytics systems.

\subsection{Mental Models of Data Augmentation}

Knowledge graphs can be difficult to reason about, especially when they are used to find data corresponding to an entire column of a dataset.  In Auger\xspace, we address this by showing the user previews of the joined dataset, as well as visualizations of the distribution of joined data and an example of the portion of the knowledge graph that is used to construct a single value (see Figure~\ref{fig:through_join}).  This approach was based on our conversations with designers of visual analytics applications for building predictive models.  A better understanding of the user's mental model of the knowledge graph and the joining process is needed to generalize this approach to other usage scenarios.  We generally aimed to hide the complexity of the underlying knowledge graph, but it may be the case that more direct exploration of the knowledge graph is helpful for some cases.

\subsection{Sparsity in Knowledge Graphs}

\label{sec:sparsity}

In this work, we rely on the high quality of Wikidata to find attributes that have sufficient support for the user's dataset.  But in many cases, only 10-15\% of the rows of a dataset will have joinable data for a given attribute.  Because knowledge graphs do not hold schematic data, they can suffer from data sparsity issues.  We feel that the experiments in this work show that there are many popular data types, including geographic entities like countries or states, that have sufficient robust data to discover attributes with full support on a user's dataset.  In situations where the attribute of interest is only present for a small percent of rows of the dataset, we suggest two possible solutions.  First, the additional data can still be useful as additional information injected into the dataset, and using imputation or setting reasonable default values (using a tool such as the \textit{data tamer}~\cite{Stonebraker2013} can make that information usable by downstream analyses.  Second, a user can explore the reduced dataset in which the sparse attribute is present, and if it results in promising analysis, they can use Auger\xspace to search for other more complete attributes that might be correlated with that sparse attribute, based on their domain expertise.  Lastly, we would like to point out that knowledge graphs are not read-only structures, and Auger\xspace may help point out where it would be valuable for an organization to invest in recording more data.

\subsection{Design Space for Interactions with Knowledge Graphs}

In Auger\xspace, the channel between the user and the knowledge graph is limited to the two subroutines shown in section~\ref{sec:technique}, which allow the user to see lists of related attributes and then construct queries given that information.  However, there is more potential to improve the user's control over the process and address edge cases by expanding the set of interactions between user and knowledge graph.  Automated processes in Auger\xspace could be replaced by collaborations between user and system.  

For example, ambiguities in the entity resolution used to retrieve a list of related attributes can be solved by user interaction.  A list of countries could refer to the governmental entities they describe, or they could refer to the national soccer teams participating in the World Cup; there will always exist cases where disambiguating the type of an attribute will require domain expertise from a user.  

The user could also benefit from more fine-grained control over the process of building aggregation queries.  Users may want to join timestamped data, which would necessitate some specification from the user of how to parse and interpret temporal columns.  Spatial data offers an additional potential, as the user might want to use geographical data in their dataset to search for the closest weather station or other geographically tagged entity.  There are many types of queries that might necessitate different user interactions than have been used in previous visual analytics systems.









\section{Conclusion}
\label{sec:conclusion}
In this work, we presented Auger\xspace, a visual analytics system for explorative information foraging using knowledge graphs.  Knowledge graphs offer a wealth of information on a broad array of topics that could be used to improve the outcome of embedded tasks of visual analytics systems.  In our usage scenarios, we demonstrated that the data gathered through Auger\xspace could result in better predictive models and better insight generation on two datasets.  And in our experiment, we showed that Auger\xspace is simple to learn and use, as all six of our participants were able to effectively explore and join data on multiple datasets within a 60 minute session.  


\acknowledgments{
We thank our collaborators in DARPA's Data Driven Discovery of Models (d3m) program, as well as the reviewers for their helpful feedback.  This work was supported by National Science Foundation grants IIS-1452977, OAC-1940175, OAC-1939945, DGE-1855886, and 1841349, as well as DARPA grant FA8750-17-2-0107.
}
\bibliographystyle{abbrv-doi}


\subsection{New Intro}



Over 20 years ago, Pirolli and Card coined the term ``information foraging'' to denote the process of seeking and gathering information to apply towards a task\cite{pirolli1999information, pirolli2005sensemaking}.  While they focused on foraging for additional entities or rows of a dataset, foraging for additional data attributes can unlock new analytical capabilities.  For example, if a dataset contains a list of countries, adding the population of those countries as \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend enables per-capita analyses. 

Traditionally, crafting new attributes and augmenting a dataset with them is done prior to any visual analysis.  Incorporating this process into visual analytics workflows can benefit both the augmentation process as well as the underlying tasks of the system in several ways.  First, the need for an additional \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend may only arise based on insights generated during analysis; if data augmentation is embedded in the system, users can toggle back and forth between analysis and augmentation.  Second, the process of discovering data and crafting a new attribute is an analytical task in its own right, and typically consists of complex querying that can be abstracted away by a visual interface.  Lastly, user-driven curation of the \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of the dataset can serve as an additional medium of communication between user and system: the user can communicate domain knowledge by adding new \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend , and can likewise learn from the attributes that are discovered by the system.

However, there are many complications in integrating data augmentation into visual analytics systems.  Finding external data and matching it to the entites in the user's dataset is nontrivial.  Once connected to external data, it can be difficult to determine which data is relevant or useful to the user's analysis.  Some attributes may require the aggregation of multiple pieces of external data and can require joining through several intermediate entities.  These hurdles should be abstracted away so that the user is not required to be a database expert.  

Large scale information repositories provide the potential for interactive data augmentation because they put the potential data at the user's fingertips. However, in order to use such repositories a number of challenges must be addressed. If the repository is in the form of a data pool (a large collection of tables, e.g. WikiTables\cite{bhagavatula2013methodswikitable}), it can be difficult to determine which tables are relevant and joinable to the user's dataset. Tabular data can often be fragmented as well: for example, if a user is foraging for data on water usage across the United States, each municipality might be responsible for publishing its own data, and the formats may not align, resulting in sparse, error-prone joins.  Joining together multiple tables can also result in ambiguities over the type of join (left, right, outer, inner) that are difficult to resolve, and limit the types of attributes that can be constructed without additional data munging.  In addition, plenty of external data may be irrelevant or even harmful to the analysis tasks at hand.  
While there have been great strides in addressing some of the technical issues in using Data Lakes (see section~\ref{sec:related_work:data_augmentation}), little work has been done to allow a user to explore potential attributes and construct new ones.  Knowledge graphs offer an alternative to data lakes \DIFdelbegin \DIFdel{, }\DIFdelend because they simplify entity matching and joining ambiguities due to their graph format.  They bring different challenges, however, in scalability and complexity, and there is still a dearth of tools available to facilitate foraging over knowledge graphs.

In this paper, we present Auger\xspace, a visual analytics system for \DIFdelbegin \DIFdel{augmenting datasets with carefully crafted columns.  These columns }\DIFdelend \DIFaddbegin \DIFadd{Columnar data Augmentation through Visual Analytics.  Carefully crafted attributes }\DIFaddend are retrieved by running queries over knowledge graphs, as illustrated in Figure~\ref{fig:teaser}\DIFaddbegin \DIFadd{, before being added to a dataset as additional columns for downstream analysis}\DIFaddend .  First, each row of a dataset is mapped to an entity in a knowledge graph. Then, to help the user identify potential information of interest, Auger\xspace explores the local graph neighborhood about entities in the dataset to determine commonly held attributes.  Using visualizations of data quality, distribution, and the local topology of the knowledge graph, Auger\xspace guides the user through crafting additional \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of data without any programming or explicit querying.  Users can express complicated operations over the knowledge graph by interacting with these visualizations. In sum, Auger\xspace abstracts away the complexity of searching, retrieving, and joining data so that the user can focus on the exploratory task of determining which attributes are relevant to their analysis.



Auger\xspace shows the promise of integrating column augmentation as a foraging process within a VA system.  \DIFaddbegin \DIFadd{The design process presented in this work in section~\ref{sec:system} can be useful for future visual analytics tools to adopt columnar data augmentation as part of their workflow.  }\DIFaddend We claim that a resulting VA system can make the exploration and discovery of additional attributes interactive, and let users construct complex \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend without programming.  Users are able to discover new attributes as well as create attributes they had in mind.  The construction of the augmented dataset can improve the outcome of downstream analytical tasks, such as insight generation,  predictive modeling, or any other number of tasks that rely on the presence of a robust dataset.  

We offer evidence of these claims by describing two \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend for insight generation and predictive modeling.  In the first \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate how a domain scientist's analysis can be enriched through cycles of in-situ column augmentation and analysis.  And in the second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we show how a user can craft attributes that result in significantly more accurate machine learning models.  We also conduct a preliminary user study on two datasets to assess the usability of Auger\xspace.  The results confirm that users are able to both discover and craft relevant attributes easily and quickly. 


























In this work, we present the following contributions:
\begin{itemize}[topsep=3pt, itemsep=0pt,parsep=3pt]
    \item We present a visual analytics system, Auger\xspace, for exploratory data augmentation using knowledge graphs.  \DIFaddbegin \DIFadd{We also describe the design process of using visualization as a medium for query construction and knowledge graph exploration. 
    }\DIFaddend \item We provide \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend of Auger\xspace being applied to both insight generation and predictive modeling to demonstrate the generality of our approach.    
    \item We conduct a \DIFaddbegin \DIFadd{preliminary }\DIFaddend user study to assess the \DIFdelbegin \DIFdel{ability }\DIFdelend \DIFaddbegin \DIFadd{usability }\DIFaddend of our system \DIFdelbegin \DIFdel{to join both }\DIFdelend \DIFaddbegin \DIFadd{in joining }\DIFaddend semantically meaningful external data \DIFdelbegin \DIFdel{as well as data that improves a predictive model trained on the dataset}\DIFdelend \DIFaddbegin \DIFadd{across two different tasks, offering validation of our design}\DIFaddend .
\end{itemize}





\section{Background and Related Work}\label{sec:related_work}

\subsection{Knowledge Graphs}
Relational databases are one of the most popular formats to store data because they can be implemented and queried effectively and efficiently.
However, recently knowledge graphs have gained popularity for storing very large datasets because their entity-based model is conducive to how humans think about data.
Knowledge graphs represent data entities as nodes in a graph and the edges between those nodes are relationships between those entities.
The term \textit{knowledge graph} has been loosely used to describe collections of information.
Several different definitions have been offered in recent years.
In practice the term has been used interchangeably with \textit{knowledge base} or \textit{ontology}\cite{ehrlinger2016towards}.
Structuring entities in the form of an ontology is also used to organize conceptual spaces, for example for the evaluation of visual analytics systems\cite{chen2019ontological} and visualizations that support machine learning tasks\cite{sacha2018vis4ml}.

The current popularity of knowledge graphs to store information can be traced back to 2012, when Google introduced its \emph{Knowledge Graph}.
Google's Knowledge Graph is a repository that retrieves facts about entities in search terms on their search results pages\cite{googleknowledgegraph2012, Dong2014}, and is used to power the Google Assistant\cite{googleAssistant} and Google Home voice queries\cite{googleHome}. Recent work has demonstrated that the approach can generalize to other artificial intelligence tasks such as image captioning and conversational agents\cite{huang2019knowledge,zheng2018question,hixon2015learning}.

Since 2012, Wikipedia and other Wikimedia projects have collected information into the \textit{Wikidata} knowledge graph.
It contains broad information about tens of millions of entities.
Other large public repositories exist, from DBPedia\cite{auer2007dbpedia}, a collection of data resources, to the domain-specific universal protein knowledgebase \textit{UniProt}\cite{uniprot2017uniprot} and the linguistic resource \textit{WordNet}\cite{Miller1995}.
Many of these graphs are part of the Linked Open Data Cloud\cite{linkedopendata2020}, a knowledge graph that contains information about other knowledge graphs.
All these graphs contain a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

\DIFdelbegin \DIFdel{A lot of information is available in machine-readable form in knowledge graphs.
With }\DIFdelend \DIFaddbegin \DIFadd{Knowledge graphs have been become popular in many research areas such as information retrieval, Natural Language Processing (NLP), and text analysis. For example, a number of approaches (e.g.,~\mbox
\cite{Dalton2014, liu2015, Rao2013, Raviv2016}}\hspace{0pt
) and tools (e.g.,~\mbox
\cite{Ferragina2010, Gabrilovich2013}}\hspace{0pt
) focus on text-centric information retrieval using entity links in underlying knowledge graphs. Examples of applications include the discovery of emerging entities~\mbox
\cite{Hoffart2014}}\hspace{0pt
, the structuring of contents into topics~\mbox
\cite{Balasubramanian2010}}\hspace{0pt
, extracting relations~\mbox
\cite{Mintz2009}}\hspace{0pt
, semantic search~\mbox
\cite{Berant2013, Krishnan2018, Nanni2018}}\hspace{0pt
, or predicting the missing relations between entities~\mbox
\cite{Dettmers2018, Yang2015}}\hspace{0pt
. Researchers in NLP and text analysis fields have also utilized knowledge graphs from different perspectives, e.g., for embedding entities and words into a continuous vector space~\mbox
\cite{Wang2014}}\hspace{0pt
, for extracting relation facts form text~\mbox
\cite{Riedel2010, Hoffmann2011}}\hspace{0pt
, for the generation of questions and answers pairs automatically~\mbox
\cite{Reddy2017}}\hspace{0pt
, and for parsing and interpretation of user natural language semantically~\mbox
\cite{Hakkani2014}}\hspace{0pt
.
}

\DIFadd{In our system, }\DIFaddend Auger\xspace\DIFaddbegin \DIFadd{, }\DIFaddend users can tap \DIFdelbegin \DIFdel{these }\DIFdelend knowledge repositories as a data source to augment a dataset \DIFaddbegin \DIFadd{through interactive visual support}\DIFaddend . Knowledge graphs have the advantage of providing clean, curated data which means that the need for data cleaning and matching is greatly reduced.
In addition, the entity-focused way in which knowledge graphs store information can make it easier to think about the relationship of data objects and thus helps users guide the augmentation process. 



\subsection{Data Augmentation}
\label{sec:related_work:data_augmentation}
\DIFaddbegin \DIFadd{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new attributes (columns) for the objects.
For image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.
}

\DIFadd{Approaches to add columns to a dataset typically vary in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new attributes from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}

\DIFaddend Automated augmentation of datasets is of interest to the database and the data science communities.
In the database community, the goal of data augmentation can manifest in many ways. 
For example, before augmentation can take place, the first challenge is to find datasets that are suitable for joining. 
In this scenario, sometimes referred to as a ``data lake''\cite{Miller2018}, the data joining system assumes that there is a finite number of candidate datasets, and the task is to identify which of them can be joined with a user's base dataset.
Systems such as Google Goods\cite{halevy2016goods}, Infogather\cite{Yakout2012}, Octopus\cite{cafarella2009data}\DIFaddbegin \DIFadd{, }\DIFaddend Aurum\cite{fernandez2018aurum}, and work by Sarma et al.\cite{das2012finding} examine the attributes of the candidate datasets and learn the relationship between those attributes and the attributes of the input data.

Another type of data augmentation appears in the form of entity matching. 
Entity matching refers to finding the same entity in different datasets, often using machine learning techniques. Once identified, the entity can be removed (for deduplication) or merged (for augmentation). Examples of entity matching systems include Magellan\cite{konda2016magellan}, Nadeef\cite{dallachiesa2013nadeef}, Autojoin\cite{zhu2017auto}, and work by Mudgal et al.\cite{mudgal2018deep}.

What the data joining and entity matching approaches have in common is that the systems assume little knowledge about the data. The challenge is therefore to identify the commonalities between the datasets (e.g. schema matching) to determine if the datasets or the entities within are related and therefore joinable.
In Auger\xspace, we take a different approach.
Instead of assuming little to no knowledge about the candidate data, we use a knowledge graph as the source of ``candidate data.'' 
Although knowledge graphs can be loosely structured, they contain well-defined relations between entities. 
This property of knowledge graphs allows Auger\xspace find relevant data for augmentation with certainty and ease.





\DIFdelbegin \DIFdel{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new features (columns) for the objects.
For image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.


\DIFdelend
\DIFdelbegin \DIFdel{Approaches to add columns to a dataset typically differ in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new features from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}\DIFdelend 


While automated approaches can discover joinable attributes for a given dataset, they lack semantic knowledge of expert users.
This may result in a large number of attributes that are irrelevant to the analysis problem being added to the dataset, hindering the users in further analyzing and gaining insights from the dataset.
In addition, in common one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.
Auger\xspace overcomes these issues by involving a user in the process.  Attributes are only added to the dataset if the user decides that they are helpful for their analysis.  

\begin{comment}
Consider a geospatial dataset, perhaps of temperature readings, with a column representing the country in which the reading took place.  In a broad data repository such as Wikidata, over 1200 attributes are potentially joinable to countries in just a single hop on the knowledge graph.  Joining irrelevant attributes into a dataset can complicate the downstream analytical tasks because of the curse of dimensionality.  

Even if an automated approach were to only select a manageable number of attributes to join, there may be semantic constraints that render certain attributes unusable that only a user would know about.  For example, consider an analyst who is studying housing prices in the United States.  If their task is to build a model for mortgage approvals, there may be certain attributes such as race that are considered protected characteristics that legally cannot be included in the analysis.  On the other hand, they may intentionally want to include these attributes so that they can do a post-hoc fairness analysis to see if those attributes can be predicted from the rest of the dataset~\cite{chouldechova2018frontiers}.  Using automated tools can be dangerous in many analysis environments, and we posit that a human in the loop tool like Auger\xspace mitigates those dangers. It is possible that the performance of Auger\xspace could be improved by using the data discovery system of Aurum rather than Wikidata or making use of the efficient queries from RDFFrames.  In addition, in one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.  For more details on how this is handled in Auger\xspace, see section~\ref{sec:system}.
\end{comment}



\subsection{\DIFdelbegin \DIFdel{Interactive }\DIFdelend Visual \DIFdelbegin \DIFdel{Data Curation}\DIFdelend \DIFaddbegin \DIFadd{Analytics Approaches}\DIFaddend }
\DIFaddbegin 

\DIFaddend Curating, improving, and augmenting an existing dataset \DIFdelbegin \DIFdel{is typically }\DIFdelend \DIFaddbegin \DIFadd{has been a popular research topic in visual analytics.  This can be }\DIFaddend done to aid interpretation and sensemaking\cite{cramer2017impact} during analysis.  \DIFdelbegin \DIFdel{Sometimes referred to as ``data blending'', many commercial visualization systems, such as Tableau\mbox
\cite{Tableau2020}}\hspace{0pt
, Alteryx\mbox
\cite{Alteryx2020}}\hspace{0pt
, and D:Swarm\mbox
\cite{DSwarm2020} }\hspace{0pt
support it by allowing users to join data tables.
The Google Data Studio\mbox
\cite{GoogleDataStudio2020} }\hspace{0pt
and OpenRefine\mbox
\cite{OpenRefine2020}}\hspace{0pt
, both based on the Webtables project\mbox
\cite{Cafarella2008}}\hspace{0pt
, also support querying web APIs
and integrating data from them into a table.
In contrast to our approach, none of these tools supports users with discovering new data for augmentation in a systematic way.
Users have to find additional data on their own before they can join it to the base dataset.


\DIFdelend Other visual interactive approaches allow users to \DIFdelbegin \DIFdel{use and integrate }\DIFdelend \DIFaddbegin \DIFadd{query and integrate query }\DIFaddend data from heterogeneous web \DIFaddbegin \DIFadd{or local }\DIFaddend sources.
This includes helping users extract information from textual web sources\cite{Hoeber2017}, query knowledge graphs\cite{Hoefler2014}, \DIFaddbegin \DIFadd{query large metadata-rich heterogenous data stores~\mbox
\cite{Smith2006}}\hspace{0pt
, identify relations at attribute levels in mixed data sets~\mbox
\cite{Jurgen2014}}\hspace{0pt
, analyse associated categories in large categorial data tables~\mbox
\cite{ Alsallakh2012}}\hspace{0pt
, }\DIFaddend or automatically create visual representations of information stored in knowledge graphs. VAiRoma helps users extract and combine information from Wikipedia articles to create visualizations that provide insight into historical events\cite{Cho2015}.
Vispedia\cite{Chan2008} lets users interactively collect and integrate data from Wikipedia tables to create visualizations and answer analysis questions.
\DIFaddbegin \DIFadd{Atlasify enables users to relate the query concept, corresponds to a Wikipedia article, to spatial entity in the underlying reference system, e.g., countries, political figure~\mbox
\cite{Hecht2012}}\hspace{0pt
.
}\DIFaddend All of these approaches help users access information stored in different forms, but compared to Auger\xspace do not support the process of augmenting an existing dataset with additional data.



\DIFdelbegin \DIFdel{Tools for data wrangling, such as Wrangler, help users to interactively identify data quality problems, and fix them to improve their datasets\mbox
\cite{Kandel2011wrangler}}\hspace{0pt
.
Data wrangling is an important step towards integrating data from diverse sources\mbox
\cite{Stonebraker2013}}\hspace{0pt
.  In 
\DIFdel{, we leave issues of data wrangling to future research, and focus on the pressing need for the discovery of relevant attributes and their extraction from knowledge graphs.  The resulting datasets could then be fed into a tool like Wrangler.
}\DIFdelend

\DIFdelbegin \DIFdel{Data improvement and curation tasks are also supported as part of creating and analyzing machine learning models.
Some approaches directly allow users to add and modify data by providing labels or corrections for train or test data\mbox
\cite{Heimerl2012, Legg2019, mansoor2019delfi}}\hspace{0pt
.
Other systems support identifying mislabeled\mbox
\cite{Ren2016, Alsallakh2017, strobelt2018s} }\hspace{0pt
or generally low quality instances\mbox
\cite{Muehlbacher2013, Gleicher2020} }\hspace{0pt
for a range of different model types.
In contrast, 
\DIFdel{does not help identifying single instances to modify, but helps users adding relevant high-quality data columns to the entire dataset.
}\DIFdelend

\DIFaddbegin 


\DIFaddend \begin{comment}
ML support in fixing data and labels:
- Helping users with correcting recorded and user labeled data~\cite{mansoor2019delfi}
- Boxer is relevant here, too (can help to find mislabeled instances) \cite{Gleicher2020}
- Also Heimerl~et~al.~\cite{Heimerl2012} active learning is a way to effectively label datasets (adding a new column to data items). (also AL: \cite{Legg2019})
- Squares\cite{Ren2016} helps users find mislabled instances in ML (classification) models, but does not support relabeling them directly. One use case shows how mislabeled instances ca be identified. Alsallakh~\cite{Alsallakh2017} claims that as well (but no use case).
Lot of work on finding mislabeled data.  Off the top of my head, Seq2Seq vis has debugging in the title. \cite{strobelt2018s}.
- M\"uhlbacher and Piringer~\cite{Muehlbacher2013} allow users to filter low quality instances to increase model quality.
\end{comment}





















\section{Knowledge Graphs}

\label{sec:knowledge_graphs}



\begin{comment}
Knowledge graphs are a generic data format in which entities are represented as nodes on a graph and the edges between those nodes represent relationships between those entities. They are used because of their flexibility and the ease of reading and writing information about entities.  Much of the current concept of knowledge graphs can be traced back to 2012, when Google introduced its Knowledge Graph, an information repository which retrieves facts about entities in search terms on their search results pages~\cite{googleknowledgegraph2012}.  Since then, Wikipedia and other Wikimedia projects have collected information into \textit{Wikidata}, a knowledge graph with broad information about tens of millions of entities.  Other large public repositories exist ranging from the collection of datasets in \textit{DBpedia}~\cite{auer2007dbpedia} to the universal protein knowledgebase \textit{UniProt}~\cite{uniprot2017uniprot}, and in fact, many of these knowledge graphs are linked themselves in the Linked Open Data Cloud~\cite{linkedopendata2020}.  These knowledge graphs represent a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

The term \textit{knowledge graph} has been loosely used to describe collections of information.   Several different definitions have been offered in recent years, and in practice the term has been used interchangably with \textit{knowledge base} or \textit{ontology}~\cite{ehrlinger2016towards}.
\end{comment}

We define our usage of the term, knowledge graph, in this section, and describe the sufficient characteristics of a knowledge graph that we assume for our system. We assume that knowledge graphs contain entities as nodes and express attributes of those entities as edges, connecting the source entity to either another entity (to express a type of relationship) or to a literal (such as a string, number or datetime).  The number of edges connected to a node can be a proxy for the number of attributes that are held by that entity.  The flexibility of this structure allows for complex relationships to be expressed between entities, including one-to-one (i.e. a country has one head of state), one-to-many (a country is composed of multiple municipalities), and many-to-many (countries share borders with other countries non-exclusively).


Auger\xspace requires that the connected knowledge graph has a data retrieval endpoint that can respond to simple queries about entities and their neighbors.  The connected graph must also have some sort of service to map from the values in the dataset uploaded to Auger\xspace to the entities on that knowledge graph.  For example, if a dataset with U.S. states is uploaded, there must be an existing service that maps from ``New York'' or ``NY'' to the entity in the knowledge graph corresponding to that state.  These two requirements allow Auger\xspace to connect a user's data to the knowledge graph and to retrieve relevant data for the user to forage from.


In the experiments in this paper, we connect Auger\xspace to Wikidata.  It meets both requirements listed above: it responds to the SPARQL query language so gathering neighborhood data about entities is simple, and it has a service \verb wbsearchentities \hspace{1mm}to map strings to entities on the graph.  But it also has some extra features that we take advantage of.  Wikidata contains a broad set of information making it easy to find related attributes for many different kinds of datasets. It also has additional metadata about the data in its graph including data-type information and human readable descriptions and labels for each node and edge in the graph.  Lastly, due to its popularity, there is a large amount of documentation and guidance on constructing complex queries.  

\DIFaddbegin \DIFadd{In future work, we hope to allow users to connect }Auger\xspace \DIFadd{to multiple knowledge graphs.  To limit scope in this work, we focused only on Wikidata because of its high standards for data quality.  In addition, it contains general knowledge that made it an applicable information repository for a number of different datasets and usage scenarios.  This also made it easier to recruit users for the study described in section~\ref{sec:user_study}.  While different knowledge graphs bring different challenges, particularly around data sparsity (see the discussion in section~\ref{sec:sparsity}), Wikidata provided a broad enough testbed to inform the design goals and interactions supported by }Auger\xspace\DIFadd{.
}

\DIFaddend


\section{Tasks and Goals}



Auger\xspace is developed as part of the DARPA Data-Driven Discovery \DIFdelbegin \DIFdel{and Modeling }\DIFdelend \DIFaddbegin \DIFadd{of Models }\DIFaddend (D3M) program whose goal is to develop software infrastructure and algorithms to make automated machine learning accessible to general data scientists. Inspired by the observation that both the use of exploratory visualization and the use of advanced machine learning are fundamentally limited by the input data, Auger was developed to help a data scientist craft better predictive models by foraging for additional \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to add to their dataset.

We conducted interview sessions with four teams within the DARPA D3M program developing applications for data exploration and predictive modeling. The goal of the interview sessions was to better understand how a tool like Auger\xspace can help the user to perform data augmentation for the purpose of improving machine learning model performance and accuracy.
Each of the interviewed teams was shown an early implementation of Auger\xspace that was able to search for related attributes and return a list of them, but without any visualizations. The participants were then asked about what additional \DIFaddbegin \DIFadd{system }\DIFaddend features and interactions would facilitate the discovery of the types of data that they were interested in for their applications.  
\begin{figure*}[thb]
\centering
\includegraphics[width=0.92\textwidth]{figures/full_system_light_acled_labels.png}
\caption{The \DIFdelbeginFL \DIFdelFL{full system }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{user interface of }Auger\xspace\DIFaddFL{, }\DIFaddendFL showing an analysis session on the ACLED dataset.  \textbf{A} shows the initial list of attributes in the table as well as augmented attributes, like the maximum inflation rate recorded for each country.  \DIFdelbeginFL \textbf{\DIFdelFL{B}}
\DIFdelFL{shows the }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{The }\DIFaddendFL list of attributes related to the \DIFdelbeginFL \DIFdelFL{Country }\DIFdelendFL \DIFaddbeginFL \textit{\DIFaddFL{Country}} \DIFaddendFL column \DIFdelbeginFL \DIFdelFL{as discovered from Wikidata}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{is shown in  }\textbf{\DIFaddFL{B}}\DIFaddendFL .  \DIFdelbeginFL \DIFdelFL{The }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Each related }\DIFaddendFL attribute \DIFdelbeginFL \DIFdelFL{of the returned attribute }\DIFdelendFL is \DIFdelbeginFL \DIFdelFL{encoded in }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{adorned with }\DIFaddendFL a \DIFdelbeginFL \DIFdelFL{symbol, a color, and a label on the left side of each row.  The }\DIFdelendFL donut chart \DIFdelbeginFL \DIFdelFL{in each row shows }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{showing }\DIFaddendFL the estimated \DIFdelbeginFL \DIFdelFL{amount of rows of the dataset that will have data available in the connected knowledge graph}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{join quality}\DIFaddendFL .   \DIFdelbeginFL \DIFdelFL{When a row in either of the two columns is clicked, additional metadata }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Additional details }\DIFaddendFL about \DIFdelbeginFL \DIFdelFL{that }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{any }\DIFaddendFL attribute is \DIFdelbeginFL \DIFdelFL{shown }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{available on demand }\DIFaddendFL in a popup, as seen in \textbf{C}.  As attributes are added to the dataset \DIFaddbeginFL \DIFaddFL{during the analysis session}\DIFaddendFL , a preview of the table is updated \DIFdelbeginFL \DIFdelFL{in the raw data table }\DIFdelendFL as seen in \textbf{D}.}

    \vspace{-3mm}

    \label{fig:system_overview}
\end{figure*}


\DIFaddbegin 

\subsection{\DIFadd{Task Analysis}}

\DIFaddend We distilled a list of tasks from the interviews that would enable a user to meaningfully augment their dataset with additional attributes.

\begin{enumerate}[label=\textbf{T\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{View a list of joinable attributes for any existing attribute in the dataset.}  Revealing the list of potential attributes that can be joined with a particular column in the dataset helps the user determine what external data is available for augmentation.  It is a key aspect of the exploration process in information foraging because it may reveal data that the user wasn't aware of.
   
    \item \textbf{Analyze the properties of possibly related attributes \textit{before} the join occurs:}
    Before joining a new attribute into a dataset, users will need to gather information about the new attribute to gauge the new attribute's impact on the quality of the dataset.
    For this reason, users should be provided with as much information as possible about the potentially joinable attributes before the join.
    This might include metadata, examples of the attribute, and any available information about the general distribution of the new data.  In addition, it is important to communicate whether there will be any missing values in the joined attribute.  
   

    \item \textbf{Specify aggregations:}
    The relationships between entities in the user's dataset and entities in the external data source are varied - they could be one-to-one, one-to-many, or many-to-many.
    In order to fit information encoded as plural relationships into a single row of data, aggregations must be specified.
    For example, a user might want to add the populations of a list of states into the their data. However, because there could be multiple measures of the state's population over the years, the user would need to perform an aggregation function over these results, such as \textsc{minimum}, \textsc{maximum}, or \textsc{average} depending on their analysis need.

    \item \textbf{Connect through to additional data:}
    Attributes relevant to the user's analysis goals may not always be directly encoded as a property of an entity that exists on the initial dataset.
    For example, a user may want to augment a dataset of countries with the population of each country's capital.
    This information may not be directly accessible as a property of each country in the data repository.
    Instead, population size is linked to the entity that represents the capital, which in turn is linked to the country.
    To support these cases, users should be supported to join \textit{through} intermediate attributes giving them fine-tuned control over how joins pass through multiple relationships.
   
   
   
   
   
\end{enumerate}

\DIFaddbegin \subsection{\DIFadd{Design Goals}}

\DIFadd{Based on this task analysis, we iterated through several sketches to come up with a set of design goals.  In the subsequent section, we describe a system with visualizations and interactions that address these goals.  However, this list of design goals should prove to have a broader impact than }Auger\xspace \DIFadd{alone - they can inform how data augmentation can be integrated into other visual analytics tools.
}

\begin{enumerate}[label=\textbf{G\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{\DIFadd{Attributes as first-class citizen.}}  \DIFadd{Because external data is found based on attributes in the original data (}\textbf{\DIFadd{T1}}\DIFadd{), the list of attributes is the most important data to encode.  In early sketches, we used a data table to show the user what the data looked like, and allowed users to click on column headers to view new attributes and view joined data in enriched cells, similar to the }\textit{\DIFadd{ETable}} \DIFadd{from Kahng et al~\mbox
\cite{kahng_2016}}\hspace{0pt
.  We ultimately decided that a table encoding dedicated too much space to the table formatting and data found in the table cells.  Instead, we settled on a primary view that listed all attributes of the dataset.  By viewing the list of attributes, users are able to build a story about what attributes might still be needed for their analysis.  This list can update as new attributes are foraged.  Additional details about each attribute, such as summary statistics and examples, can be provided on demand.
   
    }\item \textbf{\DIFadd{Visual exploration of joinable data:}} \DIFadd{In order to discover interesting joinable data (}\textbf{\DIFadd{T2}}\DIFadd{) and specify aggregations on that data to create new attributes (}\textbf{\DIFadd{T3}}\DIFadd{), users should be able to explore the set of joinable data.  Visual cues can help in this exploration, by encoding meaningful information about external data like the join quality or the data distribution.  As the dataset evolves, users should be given previews of the data they have added to enable iterations of exploration.
    }\item \textbf{\DIFadd{Provide visual examples for complex queries.}} \DIFadd{In practice, building complex queries that stepped through intermediate information on the knowledge graph becomes very complicated because it requires multiple aggregations over the joined data.  For these types of complex queries, visualizing the knowledge graph about one example in the data can help a user understand how they are combining and shaping the data.
}\end{enumerate}

\DIFaddend \section{Auger\xspace: A Visual Analytics System}

In this section, we describe the function and design of Auger\xspace.  \DIFdelbegin \DIFdel{We start by describing the }\DIFdelend \DIFaddbegin \DIFadd{The general workflow of }Auger\xspace \DIFadd{starts when a user uploads a tabular dataset.  The user can then search for related attributes of any column in their dataset containing entities that can be found in Wikidata.  They can augment their dataset with these related attributes by specifying aggregations.  After several iterations, if the user is satisfied with the augmented dataset, they can export their data for further analysis in other visual analytics tools.  
}

\DIFadd{In this section, we first describe the }\DIFaddend design of each component of the interface and their interactions.  Then, we \DIFdelbegin \DIFdel{give details on the implementation of the back-end connection to the Wikidata }\DIFdelend \DIFaddbegin \DIFadd{explain how these user interactions are translated into queries that can be executed over a }\DIFaddend knowledge graph.






\subsection{User Interface}

\DIFdelbegin \DIFdel{Here, we describe the general look and feel of 
\DIFdel{and go into depth about how its design enables the tasks listed above. As shown in }\DIFdelend
Figure~\ref{fig:system_overview}, Auger\xspace has the following main interface components:

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/through_join_small_with_labels.png}
    \vspace{-4mm}
    \caption{When a through join is specified, a dialog window pops up to allow the user to specify aggregations at each level of the join.  To help them specify that query to the system, Auger\xspace shows them an example of the neighborhood in the knowledge graph that is queried for a single row of the table.  In this image, the user would like to add the lowest life expectancy of any neighboring countries because they believe this will help their analysis of conflict.  And for each country, several values are available for life expectancy, so the user selects the ``max'' operator.  All six neighbors for Iraq, but only three are shown for simplicity of the illustration.  }
    \vspace{-3mm}

    \label{fig:through_join}
\end{figure}







 \vspace{3pt} \noindent \textbf{Column View:} This view shows the attributes and their data types (\DIFdelbegin \DIFdel{represented using colored icons) that are present in the loaded data (}\DIFdelend \DIFaddbegin \textbf{\DIFadd{G1}}\DIFadd{, }\DIFaddend see Figure~\ref{fig:system_overview}-A). \DIFdelbegin \DIFdel{As the user discovers new attributes, they can remove any particular column if they find it is not helpful to their analysis by a simple toggle button.  Users can click on any row of this view to see }\DIFdelend \DIFaddbegin \DIFadd{Users can view details on demand about each attribute, including }\DIFaddend the data distribution of that attribute shown as a histogram chart, \DIFdelbegin \DIFdel{along with any available meta-data describing that attribute (}\DIFdelend \DIFaddbegin \DIFadd{several example values, the type of unit (i.e. kilogram or mile, if applicable) (}\DIFaddend see Figure~\ref{fig:system_overview}-C).

 \vspace{3pt} \noindent \textbf{Related Attributes:} A user can \DIFdelbegin \DIFdel{toggle a search for a collection of }\DIFdelend \DIFaddbegin \DIFadd{search for }\DIFaddend related attributes of any \DIFdelbegin \DIFdel{string-like }\DIFdelend \DIFaddbegin \DIFadd{string }\DIFaddend attribute in the Column View\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T1}
\DIFdel{)}\DIFdelend .  Auger\xspace shows the \DIFdelbegin \DIFdel{retrieved attributes from the connected knowledge graph in a second column }\DIFdelend \DIFaddbegin \DIFadd{related attributes discovered from the knowledge graph }\DIFaddend next to the list of attributes of the current table, as seen in Figure~\ref{fig:system_overview}-B. 
\DIFdelbegin \DIFdel{Each row in this list represents a potential attribute that can be joined to the selected column.  Similar to attributes in the column view, metadata and distributions of each attribute can be viewed in a tooltip, as seen in Figure~\ref{fig:system_overview}-C.  The numbers surrounded by colored circles, also known as donut charts, represent the estimated percent of rows of the dataset that have joinable data for that attributein the knowledge graph.
}\DIFdelend
 \DIFaddbegin \DIFadd{An estimation of the join quality of each related attribute is encoded visually with a donut chart, and details on demand are also provided for each attribute, including distribution of values.
 }\DIFaddend This information can help the user identify whether a related attribute has the potential to help their analysis \textit{before} joining it to their data (\DIFdelbegin \DIFdel{Task }\DIFdelend \textbf{\DIFdelbegin \DIFdel{T2}\DIFdelend \DIFaddbegin \DIFadd{G2}\DIFaddend }).  Both the estimated probability and the estimated attribute distribution are calculated from random sample of possible joinable data, since calculating the true values would require completing the full join for every row in the dataset.  For more details on implementation techniques, see section~\ref{sec:technique}.

 \vspace{3pt} \noindent \textbf{Adding Attributes:} To add any attribute, users click the plus symbol, and Auger\xspace will show a drop-down menu revealing various \DIFdelbegin \DIFdel{join-operations users can perform to join this attribute to the data}\DIFdelend \DIFaddbegin \DIFadd{options for join operations}\DIFaddend . The available join operations correspond to the data type of the relationship.  If the relationship is one-to-one, i.e. a country has a single head of government, then a single value will be retrieved for each row of the dataset.  If the relationship is one-to-many or many-to-many, an aggregation must be specified.  For example, if the attribute is a collection of numbers, such as set of populations recorded for a country, the user can select numerical aggregations such as \DIFaddbegin \textit{\DIFadd{count}}\DIFadd{, }\DIFaddend \textit{mean}, \textit{max}, \textit{min}, \textit{sum}, or \textit{variance}\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T3}
\DIFdel{).  }\DIFdelend \DIFaddbegin \DIFadd{.  If the attribute is a datetime, the user can select from }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{max}}\DIFadd{, or }\textit{\DIFadd{min}}\DIFadd{.  If the attribute is a string, the user can select either }\textit{\DIFadd{count}} \DIFadd{or }\textit{\DIFadd{through}}\DIFadd{, a type of placeholder operation that let's the user step through this attribute to form complex multi-hop joins (more details are provided in the next bullet).  In addition to these join operations, for any type of data, the user can select to randomly sample from the collection.  This operation is useful if an attribute is generally expected to have only one value per entity, but due to data quality issues, some rows might have it recorded twice, like date of birth.  
 }\DIFaddend 

 It can be very costly to resolve all of the queries needed to join an additional attribute to an entire dataset; gathering the data from the knowledge graph requires the retrieval of many different parts of the knowledge graph \DIFdelbegin \DIFdel{rather than }\DIFdelend \DIFaddbegin \DIFadd{in contrast to }\DIFaddend the retrieval of a single column from a relational database.  Auger\xspace mitigates much of that time cost by only joining attributes for the top 10 rows of the dataset.  Users are still able to glance at the dataset preview, as seen in Figure~\ref{fig:system_overview}-D, and get an idea of the data that is being joined to the dataset \DIFaddbegin \DIFadd{(}\textbf{\DIFadd{G2}}\DIFadd{)}\DIFaddend .  This lets the user explore the available attributes rapidly.  When the exploration phase is completed, the full join can be executed.  

 \vspace{3pt} \noindent \textbf{Through Joins:} In some cases, the user may want to join to data through an intermediate attribute.  If the intermediate attribute has a one-to-one relationship with the rows of the original dataset, i.e. a country has one head of government, then the user can simply join the intermediate attribute first, and then use that as their starting point to seek more related attributes.  If the intermediate attribute is a one-to-many relationship, then the user must specify multiple levels of aggregation.  We call this type of aggregation a \textit{``multi-hop aggregation,''} since it requires aggregating data across multiple hops on the knowledge graph.  

 As an example, suppose the user wants to determine if a country has a lower life expectancy than its neighbors.  Consider how that would be calculated for a single country, Iraq.  To gather the required data from the knowledge graph, the user must first join to all bordering countries, then connect through them to reach the life expectancies of those countries.  Once all data is selected, aggregations must be specified to produce the desired value.  In Wikidata, countries have multiple life expectancies recorded, so the user has to specify that they want the maximum life expectancy recorded.  That value is then calculated for each bordering country (i.e. Iran, Turkey, Jordan, etc.).  Then, they have to specify that they want the minimum among all bordering countries.  Expressing this type of query is complicated enough to explain for a single value, let alone for an entire column.  
 \DIFaddbegin 

 \DIFaddend In Auger\xspace, the user is shown a simplified illustration of the topology of the knowledge graph to assist them in understanding the aggregations that the user must choose, as seen in Figure~\ref{fig:through_join}.  \DIFaddbegin \DIFadd{For a given row of the dataset, up to three values of the intermediate attribute and the target attribute are sampled from the knowledge graph.  The system then displays these values on a graph, representing a small slice of the knowledge graph.  Users can then dynamically set the aggregations (i.e. }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{mean}}\DIFadd{) that occur at each level of the join graph, and see the resulting value that would get joined for that row.  }\DIFaddend By viewing the values that will get \textit{passed through} the knowledge graph to ultimately calculate the attribute for one row of their data, users get reinforcement that the complex query they are building \DIFdelbegin \DIFdel{results in reasonable data.
 }\DIFdelend \DIFaddbegin \DIFadd{in }Auger\xspace \DIFadd{matches the query they are building in their head (}\textbf{\DIFadd{G3}}\DIFadd{).
 }\DIFaddend 

\DIFdelbegin
\textbf{\DIFdel{Showing the augmented data: }}
\DIFdel{When additional attributes are successfully added to the data, the column view shows a card on the right showing the list of attributes that are added to the data. Added columns are also shown as a nested list in the first card below the parent attribute through which it was retrieved from the knowledge graph, as seen in Figure~\ref{fig:system_overview}-A. If the added column is a string, it can then be used as a starting point to search for more related attributes.  In that manner, the user can reach further and further away from the initial attribute set by iteratively adding attributes As attributes are added, the table view updates with the new list of columns and shows the user example values for the top 10 rows of the dataset (see Figure \ref{fig:system_overview}-D).

}\DIFdelend




 









\subsection{Backend Implementation}

\label{sec:technique}

Here, we describe how our visual analytics system augments a dataset by translating the tasks supported by the user interface described above into valid queries over a knowledge graph.  Auger\xspace gathers data from the connected knowledge graph in two different subroutines.  In all examples in the paper, Auger\xspace connects to Wikidata using the SPARQL query language.\footnote{Example queries generated by the system for both subroutines are available in the supplemental material.}



\vspace{3pt} \noindent \textbf{Finding Related Attributes:} This subroutine receives a column of data as input \DIFdelbegin \DIFdel{(i.   e. the Country columnand all it's data, }\DIFdelend \DIFaddbegin \DIFadd{and returns a list of attributes along with assorted metadata.   Consider a dataset with a Country column, i.e. }\DIFaddend ``Germany,'' ``France,'' ``Austria,'' etc.\DIFdelbegin \DIFdel{), and returns a list of attributes along with assorted metadata.  
In }\DIFdelend \DIFaddbegin \DIFadd{.
Unlike a column in a SQL table, where entities and their relations might be expected to adhere to a schema, a list of countries in }\DIFaddend a knowledge graph \DIFdelbegin \DIFdel{, although these are all entities about a country, they }\DIFdelend might not share the same set of attributes\DIFdelbegin \DIFdel{(i. e. they might have different numbers of edges connected to the entities, and those edges might describe different types of relationships).  
}\DIFdelend \DIFaddbegin \DIFadd{. For example, in Wikidata, some Western countries, such as Germany, can have more than a thousand attributes, while other countries have only a fraction of that.  
}\DIFaddend In order to provide a user with a consistent set of attributes for these entities, we first need to find their commonalities.
As such, the primary goal of this subroutine is to return the list of attributes that are available on as many of the entities as possible.  
\DIFaddbegin 

\DIFaddend Checking the related attributes of each entity to gather this list can take prohibitively long, especially if the input data contains thousands to millions of rows.  \DIFdelbegin

\DIFdelend As an optimization step to make this subroutine support a user's interactive analysis, we employ a sampling-based technique to reduce computation time. With this optimization, some subset of the dataset is randomly sampled; \DIFdelbegin \DIFdel{empirically }\DIFdelend \DIFaddbegin \DIFadd{in our experiments }\DIFaddend we have found that 20-30 rows are sufficient \DIFaddbegin \DIFadd{to differentiate high-quality attributes from poor-quality ones}\DIFaddend . Each sampled row is mapped to an entity on the knowledge graph, and a list of related attributes for that row is retrieved.  Then, the lists for all the rows are compared, and the 50 attributes that appear on the most lists are returned.  \DIFdelbegin \DIFdel{The percent of lists that the attribute appears on is passed on as metadata, and is visually encoded in 
\DIFdel{as a colored donut chart, seen in each attribute row in Figure~\ref{fig:system_overview}-B.
    

\DIFdel{Various metadata is returned, including three examples for each attribute.  To get the data for the histograms, seen in Fig~\ref{fig:system_overview}-C, an additional query is run for each of }\DIFdelend \DIFaddbegin \DIFadd{In this query, we also gather all details of }\DIFaddend the \DIFdelbegin \DIFdel{50 related attributes to get 20-30 examples through which to get a rough estimate of the distribution of the data.This subroutine runs queries in parallel and is only limited in speed by the concurrency limits on the knowledge graph API. }\DIFdelend \DIFaddbegin \DIFadd{attributes that are displayed on demand, such as the data distribution of that attribute shown as a histogram chart, several example values, and the type of unit (i.e. kilogram or mile, if applicable).  }\DIFaddend For the types of exploration done in the user study and in examples given in this paper, retrieving the list of related attributes from Wikidata typically takes 2-5 seconds.

    \DIFaddbegin 

\DIFaddend \begin{figure*}[ht]
    \centering
    \includegraphics[width=\textwidth]{figures/use_case_1_tableau.png}
        \vspace{-3mm}

    \caption{Three steps of analysis of an armed conflicts dataset.  Visualizations show the number of records of each event type by (a) country, (b) basic form of government, and (c) government type.  Iniitally, only country is available.  Auger\xspace is used to gather basic form of government first, then, after some analysis by the user, it is used to gather the government type to better understand patterns in the event types.  Visualizations generated with Tableau. }
        \vspace{-3mm}
    \label{fig:tableau_use_case}
\end{figure*}

    
\vspace{3pt} \noindent \textbf{Materializing Joins:} This subroutine takes in join instructions and returns an augmented dataset with an additional column.  The join instructions specify the path taken through the knowledge graph, along with any aggregation functions, such as ``count'' or ``max.''  Auger\xspace builds a SPARQL query to crawl the knowledge graph to retrieve the desired data (see Fig~\ref{fig:teaser}).  The time to retrieve data scales linearly with the number of rows and is limited by the concurrency limits on the knowledge graph API. While the user is interactively exploring attributes, Auger\xspace will only materialize the join for the top 10 rows of the data table to provide the user with a preview of the table in real time (see Fig~\ref{fig:system_overview}-D).  When the foraging is done, all joins are materialized for the entire dataset (rather than just a sample of rows) in the order in which they were constructed by the user, which takes around 10-15 seconds on the datasets explored in the user study in section~\ref{sec:user_study}.




Auger\xspace is implemented with a VueJS frontend web application and a NodeJS backend to handle all asynchronous calls.  Queries are sent concurrently whenever possible.  Source code is withheld to maintain anonymity and will be released in the final publication.







\label{sec:system}


\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 1: Conflict Data}



To demonstrate the value of column augmentation  for insight generation, we present a \DIFdelbegin \DIFdel{use case }\DIFdelend \DIFaddbegin \DIFadd{usage scenario }\DIFaddend in which Auger\xspace is integrated into a typical workflow for the popular visual analytics tool, Tableau.   Suppose a political scientist wants to study the factors that lead to armed conflict.  They download the Armed Conflict Location \& Event Data Project (ACLED) dataset for April 2018, which contains hundreds of records of armed conflicts across many different countries and times\cite{raleigh2010introducing}.  Initially, they load the data into Tableau to analyze relationships between countries and event types.  They produce a heatmap, seen in Figure~\ref{fig:tableau_use_case}(a), to try to detect patterns, but find it difficult to visually group countries without more data.

They load their dataset into Auger\xspace to look for additional attributes that may be relevant to event types.  When the system is first loaded, the user sees a list of the attributes in the uploaded table in Figure~\ref{fig:system_overview}-A.  They also can see the first five rows of the data below in Figure~\ref{fig:system_overview}-D.  

The political scientist believes the type of government could be related to the event type, so they click the \textit{related attributes} button next to ``Country'' in Figure~\ref{fig:system_overview}-A.  The system returns a list of related attributes found by scraping Wikidata, as seen in Figure~\ref{fig:system_overview}-B (\textbf{T1}).  The user scans through the list of related attributes, looking for information about the type of government.  They see that Wikidata holds the \textit{basic form of government} for each country.  The tooltip for that attribute, seen in Figure~\ref{fig:system_overview}-C, provides more information to affirm that this holds the right data by showing the user examples of the attribute, like ``unitary state,'' ``federal republic,'' and ``absolute monarchy'' (\textbf{T2}).  The user adds this attribute to the dataset.  While looking for the form of government, the user also notices additional interesting \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend for analysis and joins them for future analysis, including the max \textit{inflation rate}, minimum \textit{life expectancy}, mean \textit{nominal GDP per capita}, and the mean \textit{Human Development Index} (\textbf{T3}).

The user loads this next iteration of the dataset into Tableau, and generates the same visualization but with the basic form of government replacing the country name, seen in Figure~\ref{fig:tableau_use_case}(b).  While some patterns begin to emerge, the user notes that the basic form of government is too fine-grained for the type of analysis they want to do, since most categories only have one or two countries in them.  They return to Auger\xspace to see if there is any higher-level categorization available for that \DIFdelbegin \DIFdel{column by searching for attributes off of that }\DIFdelend attribute (\textbf{T4}).  They find that Wikidata holds a \textit{subclass of} attribute for basic forms of government with examples including ``democracy,'' ``monarchy,'' ``republic,'' and add that attribute using a through join before returning to Tableau to generate the final visualization, seen in Figure~\ref{fig:tableau_use_case}(c).  Now, they can see a pattern in that conflicts concentrate on republics in the region, with democracies and monarchies having interesting patterns in their types of conflicts as well.  This \DIFdelbegin \DIFdel{use case }\DIFdelend \DIFaddbegin \DIFadd{usage scenario }\DIFaddend demonstrates how the integration of column augmentation into a traditional insight generation pipeline can unlock new analyses in-situ.

\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 2: Modeling Unemployment}

\label{sec:use_cases_ml}
\DIFdelbegin

\DIFdelFL{A modified version of 
\DIFdelFL{used for both Use Case 2 and Task 2 of the user study. Users can build predictive models as they augment their dataset to help them discover attributes that help predict a target variable.  This figure portrays the construction of regression models to predict the rate of unemployment in counties of the United States.  The history of trained models is shown at the bottom of the column view.   By mousing over a model, the user can see the five highest feature importance scores for that model, and the list of attributes used to train that model are highlighted in black.}}
   
\DIFdelend

In our second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate the value of Auger for a separate type of visual analytics task - the generation of a predictive model.  Many visual analytics systems have been constructed to enable domain experts to interact with and steer the generation of machine learning models on their data.  Augmenting a dataset with new \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend is one way in which a user can imbue domain knowledge into the modeling process.  They may know that certain attributes can help the prediction while others may mislead.  While it may seem unusual to augment a training set with additional attributes, the additional attributes found on the knowledge graph can likewise be added to any data that the resulting model is asked to predict on at test time when the model is deployed.

We modify Auger\xspace to be able to train a predictive model at any point in the data augmentation process\DIFdelbegin \DIFdel{(see Figure~\ref{fig:ml_use_case}).
}\DIFdelend \DIFaddbegin \DIFadd{.
}\DIFaddend When the user instructs Auger\xspace to build a model, Auger\xspace splits the data into a training and test set in the ratio of 0.8:0.2 and trains a random forest regressor.  For each model, Auger\xspace reports the \textit{R Squared} score on the training and test sets, as well as the \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend importance scores of the five most important \DIFdelbegin \DIFdel{features}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend .  

Consider Andy\xspace is a public policy analyst who seeks to build a regression model on the unemployment rate dataset\cite{unemployment2015} with $1200$ data samples from the year $2015$. Every row in the data is a county and contains a sparse set of attributes such as \textit{County}, \textit{State}, \textit{Month}, and the response variable \textit{Unemployment Rate}. 
Andy\xspace seeks to augment this data by adding meaningful \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to accurately predict the \textit{Unemployment Rate}. 
When Andy\xspace first loads the data in Auger\xspace, an initial model is trained with an \textit{R Squared} score of $0.646$ on training and $0.292$ on test set respectively. 
Andy\xspace seeks to improve the regression model's performance further by augmenting the base data using Auger\xspace's workflow and visual interface.
Andy\xspace searches for related attributes of the variable \textit{State} from the Column View. In response, Auger\xspace shows a list of related attributes that Andy\xspace may consider to add to the data. 
From these attributes they click on a set of interesting attributes such as \textit{shares border with}, \textit{head of government}, \textit{inflation rate}, etc. to see the distribution of values, and its metadata as a text view. 

From these set of attributes Andy\xspace thinks that the attributes \textit{Inflation Rate} and \textit{GDP per capita} are good attributes to predict the \textit{unemployment rate} and thus decides to add them to the data using the join operation \textit{Median}. They notice that the newly added \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend is displayed on the Table View. However looking at a few rows of the table Andy\xspace finds quite a few cells that are empty indicating that the joined data may have missing values. 
Nonetheless, Andy\xspace clicks a button to construct a new regression model using the augmented data.
Andy\xspace notices that the \textit{R Squared} score marginally improves (new score: $0.712$ and $0.301$ on training and test set respectively). They hover the mouse over the model metric card to see the list of ``Top $5$'' attributes with their weights utilised in the regression model. 
While Andy\xspace expected to see a substantial improvement in the model performance, 
they infer that the marginal improvement is probably due to missing values in the data after the join operation. \looseness=-1

Motivated to improve the model further, Andy\xspace searches for \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend that may directly help to predict the unemployment rate per county. Based on prior knowledge, Andy\xspace understands that the population of a state may be directly proportional to the unemployment rate, and they add that attribute. 
In the process of searching for other relevant \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend , Andy\xspace notices the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Maximum temperature}. Inquisitive to see if a temperature of a state is correlated with its \textit{unemployment rate} they add it to the table.
After constructing a new regression model, Andy\xspace notices that the \textit{R Squared} score improved substantially from $0.712$ to $0.855$ and from $0.301$ to $0.338$ for the training and test dataset respectively. Happy with the progress so far they decide to remove any \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend that may not be contributing to the prediction task. First they remove the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Month} (by triggering the slider on the Column View) and then triggers Auger\xspace to construct a new regression model. As expected Andy\xspace notices that the \textit{R Squared} score did not change. Next to investigate the contribution of the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Maximum temperature}, they remove it from the data. Andy\xspace notices that the \textit{R Squared} score of the new model dropped marginally meaning that the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend \textit{Maximum temperature} was helpful in improving the model. They add the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend back to the data.

In a short time, Andy\xspace has constructed a regression model with substantially better R2 score than the model trained on the base data.  They have also gained insight into which attributes are relevant to their modeling problem, which may help their understanding of the resulting machine learning model.



\section{Preliminary User Study}
\label{sec:user_study}
We evaluate Auger\xspace in a preliminary user study. The purpose of the evaluation is to validate Auger\xspace both in terms of its usability and its effectiveness in helping a user improve a machine learning model through data augmentation. 
Specifically, we hypothesize that:

\begin{itemize}
    \item \textbf{H1}: Auger\xspace allows users to accurately join external data given a written description of that data. 
   
   
    \item \textbf{H2}: Auger\xspace is able to help users discover additional data that can improve the predictive quality of machine learning models trained on the dataset.
    \DIFaddbegin \shenyu{I think we may still keep the H2 as what it was but soften its validation in both quantitative and qualitative analyses.}
\DIFaddend \end{itemize}

We recruited $6$ participants (3 Female, 3 Male), between the age of $23-36$. We required each participant to have at least an elementary knowledge of machine learning and data analysis. Due to COVID-19, we were not able to conduct an in-person study. Instead  we conducted an online study using Bluejeans\footnote{https://www.bluejeans.com/}. The participants interacted with Auger\xspace on their own computer while sharing their screen.
We provided the participants with a url to our system that was hosted on our local machine that is exposed using the Ngrok remote tunneling software\footnote{https://ngrok.com/}. The study took approximately $50-60$ minutes and we compensated the participants with a $\$10$ Amazon gift card.


\subsection{Study Design}
Before the study we asked participants to fill out a background information questionnaire regarding their name, age, gender, machine learning expertise and various use cases in which they use machine learning.
We began the study by showing the participants a tutorial video of Auger\xspace, explaining the workflow, interface GUI elements, and its interaction capabilities that support various join operations. Next we asked the participants to perform three tasks, the first of which was a practice task to ensure that they were 
sufficiently knowledgeable about Auger\xspace to perform the experimental trials. We proceeded to the experimental sessions only when we observed that the participants were confident and able to use Auger\xspace on their own. In the next two tasks we asked the participants: (1) To add a set of specified \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to a given dataset. These \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend can be added to the data using various join operations supported by Auger\xspace. (2) To freely augment the data such that they can improve the performance metric of a machine learning model (metric being a R Squared Score of a regression model). 
We used the Scikit Learn machine learning library \cite{scikit_ml} to construct \textit{Random Forest Regression} models, in the same manner as described in Section~\ref{sec:use_cases_ml}.
We collected the following data from each experimental trial: (1) \textit{Task competition time}, (2) \textit{Task Accuracy}, (3) \textit{Model performance metric} e.g., R Squared Score, and (4) \textit{User ratings} collected using a post-study Likert-scale questionnaire.


\subsection{Datasets}

For the tutorial video and practice task we used a U.S. Census Bureau dataset containing data about poverty in different counties around the U.S.\cite{poverty2020}.
The dataset contains $3136$ rows, and $6$ \DIFdelbegin \DIFdel{attributes such as }\DIFdelend \DIFaddbegin \DIFadd{attributes--the dataset index, }\DIFaddend \textit{FIPS}, \textit{State}, \textit{County}, \textit{RUCCode}, and the number of residents living in the county under the poverty line. 
For the first task in our experimental session we use the IMDB Movies dataset\cite{imdb2020} containing $500$ movies and $28$ attributes such as 
\textit{Director-name}, \textit{Duration}, \textit{Movie-Title}, \textit{Movie Cast Facebook Likes}, etc. The second task in the experimental session which required users to augment data and construct regression models, used the unemployment rate dataset\cite{unemployment2015}. This dataset contained $1200$ rows where each row corresponded to a county's poverty rate measured in a given month. This dataset contained only $5$ \DIFdelbegin \DIFdel{attributes such as }\DIFdelend \DIFaddbegin \DIFadd{attributes--the dataset index, }\DIFaddend \textit{Month}, \textit{State}, \textit{County}, and the \textit{Unemployment-Rate} (dependent variable). 
\DIFaddbegin \shenyu{As we are running out of space, maybe we should remove the detailed attribute list for each dataset and provide them as part of the supplemental materials.}
\DIFaddend

\subsection{Tasks and procedure}
Participants were first asked to complete a practice task.  
During the practice task, their answers were not recorded and their performance was not included in the analysis described below.  
For the practice task, using the Poverty dataset, we asked the participants to add three \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend : (1) the area of the county, (2) the population of the state, and (3) the earliest inception date of any county that each county shared borders with. 
While the first two \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend could be retrieved using a straight-forward \textit{join by value} operation, the third \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend required the \textit{join through} operation to search for the required \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend that was one ``hop'' away in the knowledge graph. For the first experimental task using the IMDB dataset, participants were given descriptions of four attributes to add: (1) the director's country of citizenship, (2) the number of awards received by the director, (3) the number of cast members in each movie, (4) the sum of the number of awards received by all the producers of each movie.  
Note that attributes 3 and 4 required the participants to \textit{join through} several attributes.  

The final experimental task gave participants $10$ minutes 
to augment the Unemployment rate dataset with as many attributes as they'd like that they believed would aid in building an accurate regression model to predict poverty rate.  
At any point in the 10 minutes, participants could instruct Auger\xspace to build a regression model with the current dataset to give them insight into whether they were improving the predictive modeling process.  
Building a model took about 10-20 seconds, and participants would be shown the change in R Squared Score for each new model, as well as the  weights of top ``5'' \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend used in the model. 
Random Forest regression models were used because their flexibility, speed, and accuracy met the constraints of the experiment.


\subsection{Data Collection}

After all the tasks were completed, participants were asked to fill out a post-study questionnaire (using Google Forms) that included: (1) Likert-scale\DIFaddbegin \DIFadd{\mbox
\cite{likert1932technique} }\hspace{0pt
}\DIFaddend rating questions on their use of the system, (2) a NASA-TLX\cite{hart1988development} questionnaire to measure task difficulty, and (3) Open-ended descriptive questions asking users about the interface design, the workflow, and other responses related to improve the usability of the system\footnote{Pre-experiment and Post-experiment surveys are attached as supplemental material}. 
The Likert-scale questions asked the participants to rate (in a scale of $1-7$, $1$ being ``strongly disagree'') if they found the system: (1) Easy to learn, (2) Intuitive, and (3) Expressive for data augmentation. 
We used the open-ended descriptive feedback to qualitatively evaluate the usability and the interaction design of Auger\xspace for their tasks.
Open-ended prompts included (1) \textit{Describe your thoughts about Auger\xspace}, (2) \textit{Describe things you disliked about the system or the workflow. Elaborate on how you think it could have been improved}, and (3) \textit{Describe your strategy or your process to augment the data for Tasks 1 and 2}\DIFaddbegin \DIFadd{.
}\DIFaddend With the consent of the participants,
each session was \DIFaddbegin \DIFadd{both }\DIFaddend video and audio-recorded. We encouraged the participants to verbalize their thoughts following a think-aloud protocol.
\DIFaddbegin \shenyu{Same here. As we are running out of space, maybe we should remove the details of the Likert-scale questions and open-ended prompts and provide them as part of the supplemental materials.}
\DIFaddend Furthermore, to assess if data augmentation using Auger\xspace led to any change in the regression models performance\DIFaddbegin \DIFadd{, }\DIFaddend we saved: (1) model metric (i.e., R Squared Score), (2) \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend weights, and (3) predicted values from the model. We also saved user mouse-clicks to analyze the set of \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend the user explored to augment the data.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/questionnaire_responses_2.pdf}
    \vspace{-8mm}
    \caption{Participants' responses about the system (Qn1--Qn4, the higher the better) and the overall effort (Qn5--Qn9, the lower the better).}
        \vspace{-3mm}

    \label{fig:survey_ratings}
\end{figure}


\subsection{Result and Analysis}
We report three types of results: task performance, user satisfaction, and qualitative feedback.

\subsubsection{Task Performance}

To assess \textbf{H1}, we reviewed the number of attributes that participants correctly joined in Task 1.
Five out of six participants properly joined all four expected attributes.
Only one participants missed one attribute when joining the number of awards received by the director.
This indicates that in general participants were able to use Auger\xspace to accurately add new data attributes.

Next, to assess \textbf{H2}, we similarly reviewed the \textit{R Squared} score improvement that participants achieved by augmenting additional data in Task 2.
On average, participants improved the \textit{R Squared} Score by $0.048$ ($\sigma=0.014$), starting with a baseline \textit{R Squared} Score of $0.292$. 
The area and population of the county are two most frequently joined attributes across all the participants, which is reasonable as these two attributes are usually the most influential for predicting unemployment rate. 

As a result of our analysis, we \textbf{accept \DIFdelbegin \DIFdel{both }\DIFdelend H1\DIFdelbegin \DIFdel{and H2}\DIFdelend } because all participants were able to successfully use Auger\xspace to explore and identify the correct attributes for data joining in Task \DIFdelbegin \DIFdel{1, and to improve the }\DIFdelend \DIFaddbegin \DIFadd{1.
However, we }\textbf{\DIFadd{reject H2}} \DIFadd{as the study results have no clear evidence of improving the }\DIFaddend predictive quality of a regression model in Task \DIFdelbegin \DIFdel{1. 
}\DIFdelend \DIFaddbegin \DIFadd{2.
}\shenyu{I am not sure whether we should say \textbf{accept part of H2}. By using Auger\xspace, the participants do be able to discover additional useful attributes. However, the improvement of the machine learning model (for our case, predictive quality) is not obvious, which might be restricted by the chosen datasets as a limitation of our user study.}
\DIFaddend 



\subsubsection{User satisfaction}

We use the post-study questionnaire to assess user satisfaction of the data augmentation process in Auger\xspace.
\autoref{fig:survey_ratings} shows the results of the questionnaire responses.
Although the participant size is too small to infer statistical significance using quantitative analysis, we do observe that participants found Auger\xspace easy and intuitive to use based on the likert scale user ratings (\autoref{fig:survey_ratings}, \textbf{Qn1,2}).
Participants were also satisfied with the two main functionalities provided by Auger\xspace, finding relevant attributes from the data and joining additional data (\autoref{fig:survey_ratings}, \textbf{Qn3,4}).
The mean ratings of \textbf{Qn1--4} were all 6 or above.
Furthermore, in analyzing user satisfaction related to the overall process (\autoref{fig:survey_ratings}, \textbf{Qn5--9}), we found that participants were generally satisfied with the process of conducting the tasks, as the mean ratings of \textbf{Qn5--9} are all 2 or lower.


\subsubsection{Qualitative Feedback}

To assess \textbf{H1} and \textbf{H2} from a qualitative perspective, we analysed participants’ descriptive feedback collected from post-study interviews. We also observed participants' workflow in using Auger\xspace from audio and video recordings of their computer.
We report the following three main qualitative user responses from the study:

\vspace{3pt} \noindent \textbf{Intuitive workflow:}
All the participants \DIFdelbegin \DIFdel{fount }\DIFdelend \DIFaddbegin \DIFadd{found }\DIFaddend Auger\xspace's workflow intuitive to find and augment relative data attributes.
The primary justification for Auger\xspace's intuitiveness is that Auger\xspace provides a visual interface that is easy to infer and for users to search and add relevant attributes from the data.
As \textit{P6} noted, ``\textit{I liked the interface of this tool because it was easy to navigate and add/remove columns to the base data}.''
The intuitiveness helped most participants to easily perform the requested tasks, especially Task 1.
For example, \textit{P1} described his experience in doing Task 1 as ``\textit{just looked at the question and implemented it}'' and ``\textit{it was pretty straight forward}.''

\vspace{3pt} \noindent \textbf{Two major strategies in model construction:}
When conducting Task 2, we observed two common strategies used by the participants: (1) searching for relevant attributes based on existing or prior knowledge and (2) exploring all possible combinations of available attributes to find one that results in improvement in model performance.
Some participants relied on their existing domain knowledge for augmenting the data with new attributes.
This was summarized nicely by \textit{P5}, ``\textit{I was actually using my domain knowledge. Thinking which factors can improve unemployment rate predictions, and then tried to select the variables based on the choices I had. This actually helped me improve the model's performance}.''
However, a few participants who may lack prior knowledge about the data, instead explored the dataset and tried out all possible combinations of attributes to find a model that is a significant improvement over the model trained on the base data.
``\textit{I selected columns one by one to see the change of R-squared score. By doing so, I was able to filter the variables that decrease R-square. In the end, I was able to get a relatively high R-squared model}'' (\textit{P6}).
The rest of the participants adopted a combination of both strategies.
For example, \textit{P2} described his strategies as follows, ``\textit{Thought a bit about what could boost performance, then I did some trial and error (manual feed-forward selection of attributes)}.''


\vspace{3pt} \noindent \textbf{Need for more powerful system features for expert users:}
Participants with visual analytics backgrounds expected editable visualizations to represent or encode \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend distribution differently.
``\textit{It would be cool to add additional ways of visualizing attributes}'' (\textit{P2}).
What's more, participants with database backgrounds wanted more details about queries of fetching data to increase flexibility.
``\textit{Maybe consider providing the real query of fetching the dataset to the expert users to give them more clear sense and allow them to change the query}” (\textit{P1}).
Similarly, participants with more expertise in machine learning sought more control over the model building process.
``\textit{May be add hyperparameter tuning for the model building phase, such as L1, L2 norm or change learning algorithm of the model}'' (\textit{P2}).
All these suggestions are valuable for guiding our further improvement in generalizing Auger\xspace to tackle various problem domains.



\subsection{Limitations}

While our user study results \DIFdelbegin \DIFdel{support both }\DIFdelend \textbf{\DIFaddbegin \DIFadd{support }\DIFaddend H1} \DIFdelbegin \DIFdel{and }\DIFdelend \DIFaddbegin \DIFadd{but }\DIFaddend \textbf{\DIFaddbegin \DIFadd{reject }\DIFaddend H2}, the results should be read in the light of the study limitations.
The number of the participants for our study are limited, which might be a confounding point of the results.
As we conducted the study online, the uncertainty existing in the online setup, such as internet connection, might also confound the results.
To improve the performance of the system and the fluidity of user interaction, we limited the number of related attributes that participants could fetch for each parent attribute.
Although it did help participants with a smoother user experience during the study, participants also complained about the restriction as it limited their performance of tasks, especially Task 2.
\DIFaddbegin \shenyu{While Auger\xspace does help participants discover additional useful attributes in Task2, the short task time and the limited number of accessible related attributes might confounded the results.}
\DIFaddend
Despite the limitations, the user study does help us to better identify the limitations in our system design and improve the system with new features.





\section{Discussion}
\label{sec:discussion}

Through our user study and \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we have demonstrated the efficacy of information foraging using knowledge graphs within visual analytics systems.  The generality of Auger\xspace suggests that data augmentation could \DIFaddbegin \DIFadd{be }\DIFaddend added into traditional visual analytics system workflows to improve the outcome of any embedded task.  Many fruitful avenues of research arise when considering how to apply information foraging to the full space of \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend which are served by visual analytics systems.

\subsection{Mental Models of Data Augmentation}

Knowledge graphs can be difficult to reason about, especially when they are used to find data corresponding to an entire column of a dataset.  In Auger\xspace, we address this by showing the user previews of the joined dataset, as well as visualizations of the distribution of joined data and an example of the portion of the knowledge graph that is used to construct a single value (see Figure~\ref{fig:through_join}).  This approach was based on our conversations with designers of visual analytics applications for building predictive models.  A better understanding of the user's mental model of the knowledge graph and the joining process is needed to generalize this approach to other \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend .  We generally aimed to hide the complexity of the underlying knowledge graph, but it may be the case that more direct exploration of the knowledge graph is helpful for some cases.

\DIFaddbegin \subsection{\DIFadd{Sparsity in Knowledge Graphs}}

\label{sec:sparsity}

\DIFadd{In this work, we rely on the high quality of Wikidata to find attributes that have sufficient support for the user's dataset.  But in many cases, only 10-15\% of the rows of a dataset will have joinable data for a given attribute.  Because knowledge graphs do not hold schematic data, they can suffer from data sparsity issues.  We feel that the experiments in this work show that there are many popular data types, including geographic entities like countries or states, that have sufficient robust data to discover attributes with full support on a user's dataset.  In situations where the attribute of interest is only present for a small percent of rows of the dataset, we suggest two possible solutions.  First, the additional data can still be useful as additional information injected into the dataset, and using imputation or setting reasonable default values (using a tool such as the }\textit{\DIFadd{data tamer}} \DIFadd{~\mbox
\cite{Stonebraker2013} }\hspace{0pt
can make that information usable by downstream analyses.  Second, a user can explore the reduced dataset in which the sparse attribute is present, and if it results in promising analysis, they can use }Auger\xspace \DIFadd{to search for other more complete attributes that might be correlated with that sparse attribute, based on their domain expertise.  Lastly, we would like to point out that knowledge graphs are not read-only structures, and }Auger\xspace \DIFadd{may help point out where it would be valuable for an organization to invest in recording more data.
}

\DIFaddend \subsection{Design Space for Interactions with Knowledge Graphs}

In Auger\xspace, the channel between the user and the knowledge graph is limited to the two subroutines shown in section~\ref{sec:technique}, which allow the user to see lists of related attributes and then construct queries given that information.  However, there is more potential to improve the user's control over the process and address edge cases by expanding the set of interactions between user and knowledge graph.  Automated processes in Auger\xspace could be replaced by collaborations between user and system.  

For example, ambiguities in the entity resolution used to retrieve a list of related attributes can be solved by user interaction.  A list of countries could refer to the governmental entities they describe, or they could refer to the national soccer teams participating in the World Cup; there will always exist cases where disambiguating the type of \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend will require domain expertise from a user.  

The user could also benefit from more fine-grained control over the process of building aggregation queries.  Users may want to join timestamped data, which would necessitate some specification from the user of how to parse and interpret temporal columns.  Spatial data offers an additional potential, as the user might want to use geographical data in their dataset to search for the closest weather station or other geographically tagged entity.  There are many types of queries that might necessitate different user interactions than have been used in previous visual analytics systems.


\subsection{Scalability}

In order for Auger\xspace to operate at the speed necessary for exploration, it makes use of only samples of the data, rather than the entire dataset .  When gathering the list of related attributes for a column, it samples only a subset of values from that column.  Those related attributes are then shown with estimations of their distribution as well as what percent of rows have a joinable value.  While we use a sampling-based approach, there is plenty of opportunity to develop methods that provide better estimates or allow for joining to more data in less time.  
The database community is working on tools for query optimization over knowledge graphs\cite{Mohamed2020}, which will help.  But specific indices, query optimizations, and data structures should be explored by the visual analytics community, the same way that many advancements in querying tabular data have been made to allow for faster data exploration\cite{lins2013nanocubes, wang2018neuralcubes, pahins2016hashedcubes, liu2013immens, tao2019kyrix}.






\section{Conclusion}
\label{sec:conclusion}
In this work, we presented Auger\xspace, a visual analytics system for explorative information foraging using knowledge graphs.  Knowledge graphs offer a wealth of information on a broad array of topics that could be used to improve the outcome of embedded tasks of visual analytics systems.  In our \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we demonstrated that the data gathered through Auger\xspace could result in better predictive models and better insight generation on two datasets.  And in our experiment, we showed that Auger\xspace is simple to learn and use, as all six of our participants were able to effectively explore and join data on multiple datasets within a 60 minute session.  


\acknowledgments{
We thank our collaborators in DARPA's Data Driven Discovery of Models (d3m) program.  This work was supported by National Science Foundation grants IIS-1452977, 1162037, and 1841349, as well as DARPA grant FA8750-17-2-0107.
}

\bibliographystyle{abbrv-doi}


 \DIFdelbegin
\DIFdelend\end{document}


\subsection{New Intro}



Over 20 years ago, Pirolli and Card coined the term ``information foraging'' to denote the process of seeking and gathering information to apply towards a task\cite{pirolli1999information, pirolli2005sensemaking}.  While they focused on foraging for additional entities or rows of a dataset, foraging for additional data attributes can unlock new analytical capabilities.  For example, if a dataset contains a list of countries, adding the population of those countries as \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend enables per-capita analyses. 

Traditionally, crafting new attributes and augmenting a dataset with them is done prior to any visual analysis.  Incorporating this process into visual analytics workflows can benefit both the augmentation process as well as the underlying tasks of the system in several ways.  First, the need for an additional \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend may only arise based on insights generated during analysis; if data augmentation is embedded in the system, users can toggle back and forth between analysis and augmentation.  Second, the process of discovering data and crafting a new attribute is an analytical task in its own right, and typically consists of complex querying that can be abstracted away by a visual interface.  Lastly, user-driven curation of the \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of the dataset can serve as an additional medium of communication between user and system: the user can communicate domain knowledge by adding new \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend , and can likewise learn from the attributes that are discovered by the system.

However, there are many complications in integrating data augmentation into visual analytics systems.  Finding external data and matching it to the entites in the user's dataset is nontrivial.  Once connected to external data, it can be difficult to determine which data is relevant or useful to the user's analysis.  Some attributes may require the aggregation of multiple pieces of external data and can require joining through several intermediate entities.  These hurdles should be abstracted away so that the user is not required to be a database expert.  

Large scale information repositories provide the potential for interactive data augmentation because they put the potential data at the user's fingertips. However, in order to use such repositories a number of challenges must be addressed. If the repository is in the form of a data pool (a large collection of tables, e.g. WikiTables\cite{bhagavatula2013methodswikitable}), it can be difficult to determine which tables are relevant and joinable to the user's dataset. Tabular data can often be fragmented as well: for example, if a user is foraging for data on water usage across the United States, each municipality might be responsible for publishing its own data, and the formats may not align, resulting in sparse, error-prone joins.  Joining together multiple tables can also result in ambiguities over the type of join (left, right, outer, inner) that are difficult to resolve, and limit the types of attributes that can be constructed without additional data munging.  In addition, plenty of external data may be irrelevant or even harmful to the analysis tasks at hand.  
While there have been great strides in addressing some of the technical issues in using Data Lakes (see section~\ref{sec:related_work:data_augmentation}), little work has been done to allow a user to explore potential attributes and construct new ones.  Knowledge graphs offer an alternative to data lakes \DIFdelbegin \DIFdel{, }\DIFdelend because they simplify entity matching and joining ambiguities due to their graph format.  They bring different challenges, however, in scalability and complexity, and there is still a dearth of tools available to facilitate foraging over knowledge graphs.

In this paper, we present Auger\xspace, a visual analytics system for \DIFdelbegin \DIFdel{augmenting datasets with carefully crafted columns.  These columns }\DIFdelend \DIFaddbegin \DIFadd{Columnar data Augmentation through Visual Analytics.  Carefully crafted attributes }\DIFaddend are retrieved by running queries over knowledge graphs, as illustrated in Figure~\ref{fig:teaser}\DIFaddbegin \DIFadd{, before being added to a dataset as additional columns for downstream analysis}\DIFaddend .  First, each row of a dataset is mapped to an entity in a knowledge graph. Then, to help the user identify potential information of interest, Auger\xspace explores the local graph neighborhood about entities in the dataset to determine commonly held attributes.  Using visualizations of data quality, distribution, and the local topology of the knowledge graph, Auger\xspace guides the user through crafting additional \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of data without any programming or explicit querying.  Users can express complicated operations over the knowledge graph by interacting with these visualizations. In sum, Auger\xspace abstracts away the complexity of searching, retrieving, and joining data so that the user can focus on the exploratory task of determining which attributes are relevant to their analysis.



Auger\xspace shows the promise of integrating column augmentation as a foraging process within a VA system.  \DIFaddbegin \DIFadd{The design process presented in this work in section~\ref{sec:system} can be useful for future visual analytics tools to adopt columnar data augmentation as part of their workflow.  }\DIFaddend We claim that a resulting VA system can make the exploration and discovery of additional attributes interactive, and let users construct complex \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend without programming.  Users are able to discover new attributes as well as create attributes they had in mind.  The construction of the augmented dataset can improve the outcome of downstream analytical tasks, such as insight generation,  predictive modeling, or any other number of tasks that rely on the presence of a robust dataset.  

We offer evidence of these claims by describing two \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend for insight generation and predictive modeling.  In the first \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate how a domain scientist's analysis can be enriched through cycles of in-situ column augmentation and analysis.  And in the second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we show how a user can craft attributes that result in significantly more accurate machine learning models.  We also conduct a preliminary user study on two datasets to assess the usability of Auger\xspace.  The results confirm that users are able to both discover and craft relevant attributes easily and quickly. 


























In this work, we present the following contributions:
\begin{itemize}[topsep=3pt, itemsep=0pt,parsep=3pt]
    \item We present a visual analytics system, Auger\xspace, for exploratory data augmentation using knowledge graphs.  \DIFaddbegin \DIFadd{We also describe the design process of using visualization as a medium for query construction and knowledge graph exploration. 
    }\DIFaddend \item We provide \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend of Auger\xspace being applied to both insight generation and predictive modeling to demonstrate the generality of our approach.    
    \item We conduct a \DIFaddbegin \DIFadd{preliminary }\DIFaddend user study to assess the \DIFdelbegin \DIFdel{ability }\DIFdelend \DIFaddbegin \DIFadd{usability }\DIFaddend of our system \DIFdelbegin \DIFdel{to join both }\DIFdelend \DIFaddbegin \DIFadd{in joining }\DIFaddend semantically meaningful external data \DIFdelbegin \DIFdel{as well as data that improves a predictive model trained on the dataset}\DIFdelend \DIFaddbegin \DIFadd{across two different tasks, offering validation of our design}\DIFaddend .
\end{itemize}





\section{Background and Related Work}\label{sec:related_work}

\subsection{Knowledge Graphs}
Relational databases are one of the most popular formats to store data because they can be implemented and queried effectively and efficiently.
However, recently knowledge graphs have gained popularity for storing very large datasets because their entity-based model is conducive to how humans think about data.
Knowledge graphs represent data entities as nodes in a graph and the edges between those nodes are relationships between those entities.
The term \textit{knowledge graph} has been loosely used to describe collections of information.
Several different definitions have been offered in recent years.
In practice the term has been used interchangeably with \textit{knowledge base} or \textit{ontology}\cite{ehrlinger2016towards}.
Structuring entities in the form of an ontology is also used to organize conceptual spaces, for example for the evaluation of visual analytics systems\cite{chen2019ontological} and visualizations that support machine learning tasks\cite{sacha2018vis4ml}.

The current popularity of knowledge graphs to store information can be traced back to 2012, when Google introduced its \emph{Knowledge Graph}.
Google's Knowledge Graph is a repository that retrieves facts about entities in search terms on their search results pages\cite{googleknowledgegraph2012, Dong2014}, and is used to power the Google Assistant\cite{googleAssistant} and Google Home voice queries\cite{googleHome}. Recent work has demonstrated that the approach can generalize to other artificial intelligence tasks such as image captioning and conversational agents\cite{huang2019knowledge,zheng2018question,hixon2015learning}.

Since 2012, Wikipedia and other Wikimedia projects have collected information into the \textit{Wikidata} knowledge graph.
It contains broad information about tens of millions of entities.
Other large public repositories exist, from DBPedia\cite{auer2007dbpedia}, a collection of data resources, to the domain-specific universal protein knowledgebase \textit{UniProt}\cite{uniprot2017uniprot} and the linguistic resource \textit{WordNet}\cite{Miller1995}.
Many of these graphs are part of the Linked Open Data Cloud\cite{linkedopendata2020}, a knowledge graph that contains information about other knowledge graphs.
All these graphs contain a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

\DIFdelbegin \DIFdel{A lot of information is available in machine-readable form in knowledge graphs.
With }\DIFdelend \DIFaddbegin \DIFadd{Knowledge graphs have been become popular in many research areas such as information retrieval, Natural Language Processing (NLP), and text analysis. For example, a number of approaches (e.g.,~\mbox
\cite{Dalton2014, liu2015, Rao2013, Raviv2016}}\hspace{0pt
) and tools (e.g.,~\mbox
\cite{Ferragina2010, Gabrilovich2013}}\hspace{0pt
) focus on text-centric information retrieval using entity links in underlying knowledge graphs. Examples of applications include the discovery of emerging entities~\mbox
\cite{Hoffart2014}}\hspace{0pt
, the structuring of contents into topics~\mbox
\cite{Balasubramanian2010}}\hspace{0pt
, extracting relations~\mbox
\cite{Mintz2009}}\hspace{0pt
, semantic search~\mbox
\cite{Berant2013, Krishnan2018, Nanni2018}}\hspace{0pt
, or predicting the missing relations between entities~\mbox
\cite{Dettmers2018, Yang2015}}\hspace{0pt
. Researchers in NLP and text analysis fields have also utilized knowledge graphs from different perspectives, e.g., for embedding entities and words into a continuous vector space~\mbox
\cite{Wang2014}}\hspace{0pt
, for extracting relation facts form text~\mbox
\cite{Riedel2010, Hoffmann2011}}\hspace{0pt
, for the generation of questions and answers pairs automatically~\mbox
\cite{Reddy2017}}\hspace{0pt
, and for parsing and interpretation of user natural language semantically~\mbox
\cite{Hakkani2014}}\hspace{0pt
.
}

\DIFadd{In our system, }\DIFaddend Auger\xspace\DIFaddbegin \DIFadd{, }\DIFaddend users can tap \DIFdelbegin \DIFdel{these }\DIFdelend knowledge repositories as a data source to augment a dataset \DIFaddbegin \DIFadd{through interactive visual support}\DIFaddend . Knowledge graphs have the advantage of providing clean, curated data which means that the need for data cleaning and matching is greatly reduced.
In addition, the entity-focused way in which knowledge graphs store information can make it easier to think about the relationship of data objects and thus helps users guide the augmentation process. 



\subsection{Data Augmentation}
\label{sec:related_work:data_augmentation}
\DIFaddbegin \DIFadd{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new attributes (columns) for the objects.
For image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.
}

\DIFadd{Approaches to add columns to a dataset typically vary in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new attributes from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}

\DIFaddend Automated augmentation of datasets is of interest to the database and the data science communities.
In the database community, the goal of data augmentation can manifest in many ways. 
For example, before augmentation can take place, the first challenge is to find datasets that are suitable for joining. 
In this scenario, sometimes referred to as a ``data lake''\cite{Miller2018}, the data joining system assumes that there is a finite number of candidate datasets, and the task is to identify which of them can be joined with a user's base dataset.
Systems such as Google Goods\cite{halevy2016goods}, Infogather\cite{Yakout2012}, Octopus\cite{cafarella2009data}\DIFaddbegin \DIFadd{, }\DIFaddend Aurum\cite{fernandez2018aurum}, and work by Sarma et al.\cite{das2012finding} examine the attributes of the candidate datasets and learn the relationship between those attributes and the attributes of the input data.

Another type of data augmentation appears in the form of entity matching. 
Entity matching refers to finding the same entity in different datasets, often using machine learning techniques. Once identified, the entity can be removed (for deduplication) or merged (for augmentation). Examples of entity matching systems include Magellan\cite{konda2016magellan}, Nadeef\cite{dallachiesa2013nadeef}, Autojoin\cite{zhu2017auto}, and work by Mudgal et al.\cite{mudgal2018deep}.

What the data joining and entity matching approaches have in common is that the systems assume little knowledge about the data. The challenge is therefore to identify the commonalities between the datasets (e.g. schema matching) to determine if the datasets or the entities within are related and therefore joinable.
In Auger\xspace, we take a different approach.
Instead of assuming little to no knowledge about the candidate data, we use a knowledge graph as the source of ``candidate data.'' 
Although knowledge graphs can be loosely structured, they contain well-defined relations between entities. 
This property of knowledge graphs allows Auger\xspace find relevant data for augmentation with certainty and ease.





\DIFdelbegin \DIFdel{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new features (columns) for the objects.
For image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.


\DIFdelend
\DIFdelbegin \DIFdel{Approaches to add columns to a dataset typically differ in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new features from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}\DIFdelend 


While automated approaches can discover joinable attributes for a given dataset, they lack semantic knowledge of expert users.
This may result in a large number of attributes that are irrelevant to the analysis problem being added to the dataset, hindering the users in further analyzing and gaining insights from the dataset.
In addition, in common one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.
Auger\xspace overcomes these issues by involving a user in the process.  Attributes are only added to the dataset if the user decides that they are helpful for their analysis.  

\begin{comment}
Consider a geospatial dataset, perhaps of temperature readings, with a column representing the country in which the reading took place.  In a broad data repository such as Wikidata, over 1200 attributes are potentially joinable to countries in just a single hop on the knowledge graph.  Joining irrelevant attributes into a dataset can complicate the downstream analytical tasks because of the curse of dimensionality.  

Even if an automated approach were to only select a manageable number of attributes to join, there may be semantic constraints that render certain attributes unusable that only a user would know about.  For example, consider an analyst who is studying housing prices in the United States.  If their task is to build a model for mortgage approvals, there may be certain attributes such as race that are considered protected characteristics that legally cannot be included in the analysis.  On the other hand, they may intentionally want to include these attributes so that they can do a post-hoc fairness analysis to see if those attributes can be predicted from the rest of the dataset~\cite{chouldechova2018frontiers}.  Using automated tools can be dangerous in many analysis environments, and we posit that a human in the loop tool like Auger\xspace mitigates those dangers. It is possible that the performance of Auger\xspace could be improved by using the data discovery system of Aurum rather than Wikidata or making use of the efficient queries from RDFFrames.  In addition, in one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.  For more details on how this is handled in Auger\xspace, see section~\ref{sec:system}.
\end{comment}



\subsection{\DIFdelbegin \DIFdel{Interactive }\DIFdelend Visual \DIFdelbegin \DIFdel{Data Curation}\DIFdelend \DIFaddbegin \DIFadd{Analytics Approaches}\DIFaddend }
\DIFaddbegin 

\DIFaddend Curating, improving, and augmenting an existing dataset \DIFdelbegin \DIFdel{is typically }\DIFdelend \DIFaddbegin \DIFadd{has been a popular research topic in visual analytics.  This can be }\DIFaddend done to aid interpretation and sensemaking\cite{cramer2017impact} during analysis.  \DIFdelbegin \DIFdel{Sometimes referred to as ``data blending'', many commercial visualization systems, such as Tableau\mbox
\cite{Tableau2020}}\hspace{0pt
, Alteryx\mbox
\cite{Alteryx2020}}\hspace{0pt
, and D:Swarm\mbox
\cite{DSwarm2020} }\hspace{0pt
support it by allowing users to join data tables.
The Google Data Studio\mbox
\cite{GoogleDataStudio2020} }\hspace{0pt
and OpenRefine\mbox
\cite{OpenRefine2020}}\hspace{0pt
, both based on the Webtables project\mbox
\cite{Cafarella2008}}\hspace{0pt
, also support querying web APIs
and integrating data from them into a table.
In contrast to our approach, none of these tools supports users with discovering new data for augmentation in a systematic way.
Users have to find additional data on their own before they can join it to the base dataset.


\DIFdelend Other visual interactive approaches allow users to \DIFdelbegin \DIFdel{use and integrate }\DIFdelend \DIFaddbegin \DIFadd{query and integrate query }\DIFaddend data from heterogeneous web \DIFaddbegin \DIFadd{or local }\DIFaddend sources.
This includes helping users extract information from textual web sources\cite{Hoeber2017}, query knowledge graphs\cite{Hoefler2014}, \DIFaddbegin \DIFadd{query large metadata-rich heterogenous data stores~\mbox
\cite{Smith2006}}\hspace{0pt
, identify relations at attribute levels in mixed data sets~\mbox
\cite{Jurgen2014}}\hspace{0pt
, analyse associated categories in large categorial data tables~\mbox
\cite{ Alsallakh2012}}\hspace{0pt
, }\DIFaddend or automatically create visual representations of information stored in knowledge graphs. VAiRoma helps users extract and combine information from Wikipedia articles to create visualizations that provide insight into historical events\cite{Cho2015}.
Vispedia\cite{Chan2008} lets users interactively collect and integrate data from Wikipedia tables to create visualizations and answer analysis questions.
\DIFaddbegin \DIFadd{Atlasify enables users to relate the query concept, corresponds to a Wikipedia article, to spatial entity in the underlying reference system, e.g., countries, political figure~\mbox
\cite{Hecht2012}}\hspace{0pt
.
}\DIFaddend All of these approaches help users access information stored in different forms, but compared to Auger\xspace do not support the process of augmenting an existing dataset with additional data.



\DIFdelbegin \DIFdel{Tools for data wrangling, such as Wrangler, help users to interactively identify data quality problems, and fix them to improve their datasets\mbox
\cite{Kandel2011wrangler}}\hspace{0pt
.
Data wrangling is an important step towards integrating data from diverse sources\mbox
\cite{Stonebraker2013}}\hspace{0pt
.  In 
\DIFdel{, we leave issues of data wrangling to future research, and focus on the pressing need for the discovery of relevant attributes and their extraction from knowledge graphs.  The resulting datasets could then be fed into a tool like Wrangler.
}\DIFdelend

\DIFdelbegin \DIFdel{Data improvement and curation tasks are also supported as part of creating and analyzing machine learning models.
Some approaches directly allow users to add and modify data by providing labels or corrections for train or test data\mbox
\cite{Heimerl2012, Legg2019, mansoor2019delfi}}\hspace{0pt
.
Other systems support identifying mislabeled\mbox
\cite{Ren2016, Alsallakh2017, strobelt2018s} }\hspace{0pt
or generally low quality instances\mbox
\cite{Muehlbacher2013, Gleicher2020} }\hspace{0pt
for a range of different model types.
In contrast, 
\DIFdel{does not help identifying single instances to modify, but helps users adding relevant high-quality data columns to the entire dataset.
}\DIFdelend

\DIFaddbegin 


\DIFaddend \begin{comment}
ML support in fixing data and labels:
- Helping users with correcting recorded and user labeled data~\cite{mansoor2019delfi}
- Boxer is relevant here, too (can help to find mislabeled instances) \cite{Gleicher2020}
- Also Heimerl~et~al.~\cite{Heimerl2012} active learning is a way to effectively label datasets (adding a new column to data items). (also AL: \cite{Legg2019})
- Squares\cite{Ren2016} helps users find mislabled instances in ML (classification) models, but does not support relabeling them directly. One use case shows how mislabeled instances ca be identified. Alsallakh~\cite{Alsallakh2017} claims that as well (but no use case).
Lot of work on finding mislabeled data.  Off the top of my head, Seq2Seq vis has debugging in the title. \cite{strobelt2018s}.
- M\"uhlbacher and Piringer~\cite{Muehlbacher2013} allow users to filter low quality instances to increase model quality.
\end{comment}





















\section{Knowledge Graphs}

\label{sec:knowledge_graphs}



\begin{comment}
Knowledge graphs are a generic data format in which entities are represented as nodes on a graph and the edges between those nodes represent relationships between those entities. They are used because of their flexibility and the ease of reading and writing information about entities.  Much of the current concept of knowledge graphs can be traced back to 2012, when Google introduced its Knowledge Graph, an information repository which retrieves facts about entities in search terms on their search results pages~\cite{googleknowledgegraph2012}.  Since then, Wikipedia and other Wikimedia projects have collected information into \textit{Wikidata}, a knowledge graph with broad information about tens of millions of entities.  Other large public repositories exist ranging from the collection of datasets in \textit{DBpedia}~\cite{auer2007dbpedia} to the universal protein knowledgebase \textit{UniProt}~\cite{uniprot2017uniprot}, and in fact, many of these knowledge graphs are linked themselves in the Linked Open Data Cloud~\cite{linkedopendata2020}.  These knowledge graphs represent a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

The term \textit{knowledge graph} has been loosely used to describe collections of information.   Several different definitions have been offered in recent years, and in practice the term has been used interchangably with \textit{knowledge base} or \textit{ontology}~\cite{ehrlinger2016towards}.
\end{comment}

We define our usage of the term, knowledge graph, in this section, and describe the sufficient characteristics of a knowledge graph that we assume for our system. We assume that knowledge graphs contain entities as nodes and express attributes of those entities as edges, connecting the source entity to either another entity (to express a type of relationship) or to a literal (such as a string, number or datetime).  The number of edges connected to a node can be a proxy for the number of attributes that are held by that entity.  The flexibility of this structure allows for complex relationships to be expressed between entities, including one-to-one (i.e. a country has one head of state), one-to-many (a country is composed of multiple municipalities), and many-to-many (countries share borders with other countries non-exclusively).


Auger\xspace requires that the connected knowledge graph has a data retrieval endpoint that can respond to simple queries about entities and their neighbors.  The connected graph must also have some sort of service to map from the values in the dataset uploaded to Auger\xspace to the entities on that knowledge graph.  For example, if a dataset with U.S. states is uploaded, there must be an existing service that maps from ``New York'' or ``NY'' to the entity in the knowledge graph corresponding to that state.  These two requirements allow Auger\xspace to connect a user's data to the knowledge graph and to retrieve relevant data for the user to forage from.


In the experiments in this paper, we connect Auger\xspace to Wikidata.  It meets both requirements listed above: it responds to the SPARQL query language so gathering neighborhood data about entities is simple, and it has a service \verb wbsearchentities \hspace{1mm}to map strings to entities on the graph.  But it also has some extra features that we take advantage of.  Wikidata contains a broad set of information making it easy to find related attributes for many different kinds of datasets. It also has additional metadata about the data in its graph including data-type information and human readable descriptions and labels for each node and edge in the graph.  Lastly, due to its popularity, there is a large amount of documentation and guidance on constructing complex queries.  

\DIFaddbegin \DIFadd{In future work, we hope to allow users to connect }Auger\xspace \DIFadd{to multiple knowledge graphs.  To limit scope in this work, we focused only on Wikidata because of its high standards for data quality.  In addition, it contains general knowledge that made it an applicable information repository for a number of different datasets and usage scenarios.  This also made it easier to recruit users for the study described in section~\ref{sec:user_study}.  While different knowledge graphs bring different challenges, particularly around data sparsity (see the discussion in section~\ref{sec:sparsity}), Wikidata provided a broad enough testbed to inform the design goals and interactions supported by }Auger\xspace\DIFadd{.
}

\DIFaddend


\section{Tasks and Goals}



Auger\xspace is developed as part of the DARPA Data-Driven Discovery \DIFdelbegin \DIFdel{and Modeling }\DIFdelend \DIFaddbegin \DIFadd{of Models }\DIFaddend (D3M) program whose goal is to develop software infrastructure and algorithms to make automated machine learning accessible to general data scientists. Inspired by the observation that both the use of exploratory visualization and the use of advanced machine learning are fundamentally limited by the input data, \DIFdelbegin \DIFdel{Auger }\DIFdelend \DIFaddbegin Auger\xspace \DIFaddend was developed to help a data scientist craft better predictive models by foraging for additional \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to add to their dataset.

We conducted interview sessions with four teams within the DARPA D3M program developing applications for data exploration and predictive modeling. The goal of the interview sessions was to better understand how a tool like Auger\xspace can help the user to perform data augmentation for the purpose of improving machine learning model performance and accuracy.
Each of the interviewed teams was shown an early implementation of Auger\xspace that was able to search for related attributes and return a list of them, but without any visualizations. The participants were then asked about what additional \DIFaddbegin \DIFadd{system }\DIFaddend features and interactions would facilitate the discovery of the types of data that they were interested in for their applications.  
\begin{figure*}[thb]
\centering
\includegraphics[width=0.92\textwidth]{figures/full_system_light_acled_labels.png}
\caption{The \DIFdelbeginFL \DIFdelFL{full system }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{user interface of }Auger\xspace\DIFaddFL{, }\DIFaddendFL showing an analysis session on the ACLED dataset.  \textbf{A} shows the initial list of attributes in the table as well as augmented attributes, like the maximum inflation rate recorded for each country.  \DIFdelbeginFL \textbf{\DIFdelFL{B}}
\DIFdelFL{shows the }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{The }\DIFaddendFL list of attributes related to the \DIFdelbeginFL \DIFdelFL{Country }\DIFdelendFL \DIFaddbeginFL \textit{\DIFaddFL{Country}} \DIFaddendFL column \DIFdelbeginFL \DIFdelFL{as discovered from Wikidata}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{is shown in  }\textbf{\DIFaddFL{B}}\DIFaddendFL .  \DIFdelbeginFL \DIFdelFL{The }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Each related }\DIFaddendFL attribute \DIFdelbeginFL \DIFdelFL{of the returned attribute }\DIFdelendFL is \DIFdelbeginFL \DIFdelFL{encoded in }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{adorned with }\DIFaddendFL a \DIFdelbeginFL \DIFdelFL{symbol, a color, and a label on the left side of each row.  The }\DIFdelendFL donut chart \DIFdelbeginFL \DIFdelFL{in each row shows }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{showing }\DIFaddendFL the estimated \DIFdelbeginFL \DIFdelFL{amount of rows of the dataset that will have data available in the connected knowledge graph}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{join quality}\DIFaddendFL .   \DIFdelbeginFL \DIFdelFL{When a row in either of the two columns is clicked, additional metadata }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Additional details }\DIFaddendFL about \DIFdelbeginFL \DIFdelFL{that }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{any }\DIFaddendFL attribute is \DIFdelbeginFL \DIFdelFL{shown }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{available on demand }\DIFaddendFL in a popup, as seen in \textbf{C}.  As attributes are added to the dataset \DIFaddbeginFL \DIFaddFL{during the analysis session}\DIFaddendFL , a preview of the table is updated \DIFdelbeginFL \DIFdelFL{in the raw data table }\DIFdelendFL as seen in \textbf{D}.}

    \vspace{-3mm}

    \label{fig:system_overview}
\end{figure*}


\DIFaddbegin 

\subsection{\DIFadd{Task Analysis}}

\DIFaddend We distilled a list of tasks from the interviews that would enable a user to meaningfully augment their dataset with additional attributes.

\begin{enumerate}[label=\textbf{T\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{View a list of joinable attributes for any existing attribute in the dataset.}  Revealing the list of potential attributes that can be joined with a particular column in the dataset helps the user determine what external data is available for augmentation.  It is a key aspect of the exploration process in information foraging because it may reveal data that the user wasn't aware of.
   
    \item \textbf{Analyze the properties of possibly related attributes \textit{before} the join occurs:}
    Before joining a new attribute into a dataset, users will need to gather information about the new attribute to gauge the new attribute's impact on the quality of the dataset.
    For this reason, users should be provided with as much information as possible about the potentially joinable attributes before the join.
    This might include metadata, examples of the attribute, and any available information about the general distribution of the new data.  In addition, it is important to communicate whether there will be any missing values in the joined attribute.  
   

    \item \textbf{Specify aggregations:}
    The relationships between entities in the user's dataset and entities in the external data source are varied - they could be one-to-one, one-to-many, or many-to-many.
    In order to fit information encoded as plural relationships into a single row of data, aggregations must be specified.
    For example, a user might want to add the populations of a list of states into the their data. However, because there could be multiple measures of the state's population over the years, the user would need to perform an aggregation function over these results, such as \textsc{minimum}, \textsc{maximum}, or \textsc{average} depending on their analysis need.

    \item \textbf{Connect through to additional data:}
    Attributes relevant to the user's analysis goals may not always be directly encoded as a property of an entity that exists on the initial dataset.
    For example, a user may want to augment a dataset of countries with the population of each country's capital.
    This information may not be directly accessible as a property of each country in the data repository.
    Instead, population size is linked to the entity that represents the capital, which in turn is linked to the country.
    To support these cases, users should be supported to join \textit{through} intermediate attributes giving them fine-tuned control over how joins pass through multiple relationships.
   
   
   
   
   
\end{enumerate}

\DIFaddbegin \subsection{\DIFadd{Design Goals}}

\DIFadd{Based on this task analysis, we iterated through several sketches to come up with a set of design goals.  In the subsequent section, we describe a system with visualizations and interactions that address these goals.  However, this list of design goals should prove to have a broader impact than }Auger\xspace \DIFadd{alone - they can inform how data augmentation can be integrated into other visual analytics tools.
}

\begin{enumerate}[label=\textbf{G\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{\DIFadd{Attributes as first-class citizen.}}  \DIFadd{Because external data is found based on attributes in the original data (}\textbf{\DIFadd{T1}}\DIFadd{), the list of attributes is the most important data to encode.  In early sketches, we used a data table to show the user what the data looked like, and allowed users to click on column headers to view new attributes and view joined data in enriched cells, similar to the }\textit{\DIFadd{ETable}} \DIFadd{from Kahng et al~\mbox
\cite{kahng_2016}}\hspace{0pt
.  We ultimately decided that a table encoding dedicated too much space to the table formatting and data found in the table cells.  Instead, we settled on a primary view that listed all attributes of the dataset.  By viewing the list of attributes, users are able to build a story about what attributes might still be needed for their analysis.  This list can update as new attributes are foraged.  Additional details about each attribute, such as summary statistics and examples, can be provided on demand.
   
    }\item \textbf{\DIFadd{Visual exploration of joinable data:}} \DIFadd{In order to discover interesting joinable data (}\textbf{\DIFadd{T2}}\DIFadd{) and specify aggregations on that data to create new attributes (}\textbf{\DIFadd{T3}}\DIFadd{), users should be able to explore the set of joinable data.  Visual cues can help in this exploration, by encoding meaningful information about external data like the join quality or the data distribution.  As the dataset evolves, users should be given previews of the data they have added to enable iterations of exploration.
    }\item \textbf{\DIFadd{Provide visual examples for complex queries.}} \DIFadd{In practice, building complex queries that stepped through intermediate information on the knowledge graph becomes very complicated because it requires multiple aggregations over the joined data.  For these types of complex queries, visualizing the knowledge graph about one example in the data can help a user understand how they are combining and shaping the data.
}\end{enumerate}

\DIFaddend \section{Auger\xspace: A Visual Analytics System}

In this section, we describe the function and design of Auger\xspace.  \DIFdelbegin \DIFdel{We start by describing the }\DIFdelend \DIFaddbegin \DIFadd{The general workflow of }Auger\xspace \DIFadd{starts when a user uploads a tabular dataset.  The user can then search for related attributes of any column in their dataset containing entities that can be found in Wikidata.  They can augment their dataset with these related attributes by specifying aggregations.  After several iterations, if the user is satisfied with the augmented dataset, they can export their data for further analysis in other visual analytics tools.  
}

\DIFadd{In this section, we first describe the }\DIFaddend design of each component of the interface and their interactions.  Then, we \DIFdelbegin \DIFdel{give details on the implementation of the back-end connection to the Wikidata }\DIFdelend \DIFaddbegin \DIFadd{explain how these user interactions are translated into queries that can be executed over a }\DIFaddend knowledge graph.






\subsection{User Interface}

\DIFdelbegin \DIFdel{Here, we describe the general look and feel of 
\DIFdel{and go into depth about how its design enables the tasks listed above. As shown in }\DIFdelend
Figure~\ref{fig:system_overview}, Auger\xspace has the following main interface components:

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/through_join_small_with_labels.png}
    \vspace{-4mm}
    \caption{When a through join is specified, a dialog window pops up to allow the user to specify aggregations at each level of the join.  To help them specify that query to the system, Auger\xspace shows them an example of the neighborhood in the knowledge graph that is queried for a single row of the table.  In this image, the user would like to add the lowest life expectancy of any neighboring countries because they believe this will help their analysis of conflict.  And for each country, several values are available for life expectancy, so the user selects the ``max'' operator.  All six neighbors for Iraq, but only three are shown for simplicity of the illustration.  }
    \vspace{-3mm}

    \label{fig:through_join}
\end{figure}







 \vspace{3pt} \noindent \textbf{Column View:} This view shows the attributes and their data types (\DIFdelbegin \DIFdel{represented using colored icons) that are present in the loaded data (}\DIFdelend \DIFaddbegin \textbf{\DIFadd{G1}}\DIFadd{, }\DIFaddend see Figure~\ref{fig:system_overview}-A). \DIFdelbegin \DIFdel{As the user discovers new attributes, they can remove any particular column if they find it is not helpful to their analysis by a simple toggle button.  Users can click on any row of this view to see }\DIFdelend \DIFaddbegin \DIFadd{Users can view details on demand about each attribute, including }\DIFaddend the data distribution of that attribute shown as a histogram chart, \DIFdelbegin \DIFdel{along with any available meta-data describing that attribute (}\DIFdelend \DIFaddbegin \DIFadd{several example values, the type of unit (i.e. kilogram or mile, if applicable) (}\DIFaddend see Figure~\ref{fig:system_overview}-C).

 \vspace{3pt} \noindent \textbf{Related Attributes:} A user can \DIFdelbegin \DIFdel{toggle a search for a collection of }\DIFdelend \DIFaddbegin \DIFadd{search for }\DIFaddend related attributes of any \DIFdelbegin \DIFdel{string-like }\DIFdelend \DIFaddbegin \DIFadd{string }\DIFaddend attribute in the Column View\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T1}
\DIFdel{)}\DIFdelend .  Auger\xspace shows the \DIFdelbegin \DIFdel{retrieved attributes from the connected knowledge graph in a second column }\DIFdelend \DIFaddbegin \DIFadd{related attributes discovered from the knowledge graph }\DIFaddend next to the list of attributes of the current table, as seen in Figure~\ref{fig:system_overview}-B. 
\DIFdelbegin \DIFdel{Each row in this list represents a potential attribute that can be joined to the selected column.  Similar to attributes in the column view, metadata and distributions of each attribute can be viewed in a tooltip, as seen in Figure~\ref{fig:system_overview}-C.  The numbers surrounded by colored circles, also known as donut charts, represent the estimated percent of rows of the dataset that have joinable data for that attributein the knowledge graph.
}\DIFdelend
 \DIFaddbegin \DIFadd{An estimation of the join quality of each related attribute is encoded visually with a donut chart, and details on demand are also provided for each attribute, including distribution of values.
 }\DIFaddend This information can help the user identify whether a related attribute has the potential to help their analysis \textit{before} joining it to their data (\DIFdelbegin \DIFdel{Task }\DIFdelend \textbf{\DIFdelbegin \DIFdel{T2}\DIFdelend \DIFaddbegin \DIFadd{G2}\DIFaddend }).  Both the estimated probability and the estimated attribute distribution are calculated from random sample of possible joinable data, since calculating the true values would require completing the full join for every row in the dataset.  For more details on implementation techniques, see section~\ref{sec:technique}.

 \vspace{3pt} \noindent \textbf{Adding Attributes:} To add any attribute, users click the plus symbol, and Auger\xspace will show a drop-down menu revealing various \DIFdelbegin \DIFdel{join-operations users can perform to join this attribute to the data}\DIFdelend \DIFaddbegin \DIFadd{options for join operations}\DIFaddend . The available join operations correspond to the data type of the relationship.  If the relationship is one-to-one, i.e. a country has a single head of government, then a single value will be retrieved for each row of the dataset.  If the relationship is one-to-many or many-to-many, an aggregation must be specified.  For example, if the attribute is a collection of numbers, such as set of populations recorded for a country, the user can select numerical aggregations such as \DIFaddbegin \textit{\DIFadd{count}}\DIFadd{, }\DIFaddend \textit{mean}, \textit{max}, \textit{min}, \textit{sum}, or \textit{variance}\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T3}
\DIFdel{).  }\DIFdelend \DIFaddbegin \DIFadd{.  If the attribute is a datetime, the user can select from }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{max}}\DIFadd{, or }\textit{\DIFadd{min}}\DIFadd{.  If the attribute is a string, the user can select either }\textit{\DIFadd{count}} \DIFadd{or }\textit{\DIFadd{through}}\DIFadd{, a type of placeholder operation that let's the user step through this attribute to form complex multi-hop joins (more details are provided in the next bullet).  In addition to these join operations, for any type of data, the user can select to randomly sample from the collection.  This operation is useful if an attribute is generally expected to have only one value per entity, but due to data quality issues, some rows might have it recorded twice, like date of birth.  
 }\DIFaddend 

 It can be very costly to resolve all of the queries needed to join an additional attribute to an entire dataset; gathering the data from the knowledge graph requires the retrieval of many different parts of the knowledge graph \DIFdelbegin \DIFdel{rather than }\DIFdelend \DIFaddbegin \DIFadd{in contrast to }\DIFaddend the retrieval of a single column from a relational database.  Auger\xspace mitigates much of that time cost by only joining attributes for the top 10 rows of the dataset.  Users are still able to glance at the dataset preview, as seen in Figure~\ref{fig:system_overview}-D, and get an idea of the data that is being joined to the dataset \DIFaddbegin \DIFadd{(}\textbf{\DIFadd{G2}}\DIFadd{)}\DIFaddend .  This lets the user explore the available attributes rapidly.  When the exploration phase is completed, the full join can be executed.  

 \vspace{3pt} \noindent \textbf{Through Joins:} In some cases, the user may want to join to data through an intermediate attribute.  If the intermediate attribute has a one-to-one relationship with the rows of the original dataset, i.e. a country has one head of government, then the user can simply join the intermediate attribute first, and then use that as their starting point to seek more related attributes.  If the intermediate attribute is a one-to-many relationship, then the user must specify multiple levels of aggregation.  We call this type of aggregation a \textit{``multi-hop aggregation,''} since it requires aggregating data across multiple hops on the knowledge graph.  

 As an example, suppose the user wants to determine if a country has a lower life expectancy than its neighbors.  Consider how that would be calculated for a single country, Iraq.  To gather the required data from the knowledge graph, the user must first join to all bordering countries, then connect through them to reach the life expectancies of those countries.  Once all data is selected, aggregations must be specified to produce the desired value.  In Wikidata, countries have multiple life expectancies recorded, so the user has to specify that they want the maximum life expectancy recorded.  That value is then calculated for each bordering country (i.e. Iran, Turkey, Jordan, etc.).  Then, they have to specify that they want the minimum among all bordering countries.  Expressing this type of query is complicated enough to explain for a single value, let alone for an entire column.  
 \DIFaddbegin 

 \DIFaddend In Auger\xspace, the user is shown a simplified illustration of the topology of the knowledge graph to assist them in understanding the aggregations that the user must choose, as seen in Figure~\ref{fig:through_join}.  \DIFaddbegin \DIFadd{For a given row of the dataset, up to three values of the intermediate attribute and the target attribute are sampled from the knowledge graph.  The system then displays these values on a graph, representing a small slice of the knowledge graph.  Users can then dynamically set the aggregations (i.e. }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{mean}}\DIFadd{) that occur at each level of the join graph, and see the resulting value that would get joined for that row.  }\DIFaddend By viewing the values that will get \textit{passed through} the knowledge graph to ultimately calculate the attribute for one row of their data, users get reinforcement that the complex query they are building \DIFdelbegin \DIFdel{results in reasonable data.
 }\DIFdelend \DIFaddbegin \DIFadd{in }Auger\xspace \DIFadd{matches the query they are building in their head (}\textbf{\DIFadd{G3}}\DIFadd{).
 }\DIFaddend 

\DIFdelbegin
\textbf{\DIFdel{Showing the augmented data: }}
\DIFdel{When additional attributes are successfully added to the data, the column view shows a card on the right showing the list of attributes that are added to the data. Added columns are also shown as a nested list in the first card below the parent attribute through which it was retrieved from the knowledge graph, as seen in Figure~\ref{fig:system_overview}-A. If the added column is a string, it can then be used as a starting point to search for more related attributes.  In that manner, the user can reach further and further away from the initial attribute set by iteratively adding attributes As attributes are added, the table view updates with the new list of columns and shows the user example values for the top 10 rows of the dataset (see Figure \ref{fig:system_overview}-D).

}\DIFdelend




 









\subsection{Backend Implementation}

\label{sec:technique}

Here, we describe how our visual analytics system augments a dataset by translating the tasks supported by the user interface described above into valid queries over a knowledge graph.  Auger\xspace gathers data from the connected knowledge graph in two different subroutines.  In all examples in the paper, Auger\xspace connects to Wikidata using the SPARQL query language.\footnote{Example queries generated by the system for both subroutines are available in the supplemental material.}



\vspace{3pt} \noindent \textbf{Finding Related Attributes:} This subroutine receives a column of data as input \DIFdelbegin \DIFdel{(i.   e. the Country columnand all it's data, }\DIFdelend \DIFaddbegin \DIFadd{and returns a list of attributes along with assorted metadata.   Consider a dataset with a Country column, i.e. }\DIFaddend ``Germany,'' ``France,'' ``Austria,'' etc.\DIFdelbegin \DIFdel{), and returns a list of attributes along with assorted metadata.  
In }\DIFdelend \DIFaddbegin \DIFadd{.
Unlike a column in a SQL table, where entities and their relations might be expected to adhere to a schema, a list of countries in }\DIFaddend a knowledge graph \DIFdelbegin \DIFdel{, although these are all entities about a country, they }\DIFdelend might not share the same set of attributes\DIFdelbegin \DIFdel{(i. e. they might have different numbers of edges connected to the entities, and those edges might describe different types of relationships).  
}\DIFdelend \DIFaddbegin \DIFadd{. For example, in Wikidata, some Western countries, such as Germany, can have more than a thousand attributes, while other countries have only a fraction of that.  
}\DIFaddend In order to provide a user with a consistent set of attributes for these entities, we first need to find their commonalities.
As such, the primary goal of this subroutine is to return the list of attributes that are available on as many of the entities as possible.  
\DIFaddbegin 

\DIFaddend Checking the related attributes of each entity to gather this list can take prohibitively long, especially if the input data contains thousands to millions of rows.  \DIFdelbegin

\DIFdelend As an optimization step to make this subroutine support a user's interactive analysis, we employ a sampling-based technique to reduce computation time. With this optimization, some subset of the dataset is randomly sampled; \DIFdelbegin \DIFdel{empirically }\DIFdelend \DIFaddbegin \DIFadd{in our experiments }\DIFaddend we have found that 20-30 rows are sufficient \DIFaddbegin \DIFadd{to differentiate high-quality attributes from poor-quality ones}\DIFaddend . Each sampled row is mapped to an entity on the knowledge graph, and a list of related attributes for that row is retrieved.  Then, the lists for all the rows are compared, and the 50 attributes that appear on the most lists are returned.  \DIFdelbegin \DIFdel{The percent of lists that the attribute appears on is passed on as metadata, and is visually encoded in 
\DIFdel{as a colored donut chart, seen in each attribute row in Figure~\ref{fig:system_overview}-B.
    

\DIFdel{Various metadata is returned, including three examples for each attribute.  To get the data for the histograms, seen in Fig~\ref{fig:system_overview}-C, an additional query is run for each of }\DIFdelend \DIFaddbegin \DIFadd{In this query, we also gather all details of }\DIFaddend the \DIFdelbegin \DIFdel{50 related attributes to get 20-30 examples through which to get a rough estimate of the distribution of the data.This subroutine runs queries in parallel and is only limited in speed by the concurrency limits on the knowledge graph API. }\DIFdelend \DIFaddbegin \DIFadd{attributes that are displayed on demand, such as the data distribution of that attribute shown as a histogram chart, several example values, and the type of unit (i.e. kilogram or mile, if applicable).  }\DIFaddend For the types of exploration done in the user study and in examples given in this paper, retrieving the list of related attributes from Wikidata typically takes 2-5 seconds.

\DIFdelbegin
\DIFdelendFL
    \DIFaddbeginFL 



\begin{figure}[ht]
    \DIFaddendFL \centering
    \DIFdelbeginFL
\DIFdelendFL \DIFaddbeginFL \includegraphics[width=0.45\textwidth]{figures/acled_gdp_fatalities.png}
        \DIFaddendFL \vspace{-3mm}

    \caption{\DIFdelbeginFL \DIFdelFL{Three steps of analysis of an armed conflicts dataset.  Visualizations show }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Visualizing }\DIFaddendFL the number of records \DIFdelbeginFL \DIFdelFL{of each event type }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{in the ACLED dataset in April 2018 }\DIFaddendFL by \DIFdelbeginFL \DIFdelFL{(a) country}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{per capita GDP}\DIFaddendFL , \DIFdelbeginFL \DIFdelFL{(b) basic form of government, and (c) government type}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{a field extracted from Wikidata}\DIFaddendFL .  \DIFdelbeginFL \DIFdelFL{Iniitally, only country is available.  
\DIFdelFL{is used to gather basic form }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Magnitude }\DIFaddendFL of \DIFdelbeginFL \DIFdelFL{government first}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{marks encodes number of fatalities}\DIFaddendFL , \DIFdelbeginFL \DIFdelFL{then, after some analysis by the user, it is used to gather the government type to better understand patterns in the }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{while color encodes }\DIFaddendFL event \DIFdelbeginFL \DIFdelFL{types}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{type}\DIFaddendFL .    \DIFdelbeginFL \DIFdelFL{Visualizations }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Visualization }\DIFaddendFL generated with Tableau. }
        \vspace{-3mm}
    \label{fig:tableau_use_case}
\DIFdelbeginFL
\DIFdelend \DIFaddbegin \end{figure}
\DIFaddend 

    
\vspace{3pt} \noindent \textbf{Materializing Joins:} This subroutine takes in join instructions and returns an augmented dataset with an additional column.  The join instructions specify the path taken through the knowledge graph, along with any aggregation functions, such as ``count'' or ``max.''  Auger\xspace builds a SPARQL query to crawl the knowledge graph to retrieve the desired data (see Fig~\ref{fig:teaser}).  The time to retrieve data scales linearly with the number of rows and is limited by the concurrency limits on the knowledge graph API. While the user is interactively exploring attributes, Auger\xspace will only materialize the join for the top 10 rows of the data table to provide the user with a preview of the table in real time (see Fig~\ref{fig:system_overview}-D).  When the foraging is done, all joins are materialized for the entire dataset (rather than just a sample of rows) in the order in which they were constructed by the user, which takes around 10-15 seconds on the datasets explored in the user study in section~\ref{sec:user_study}.




Auger\xspace is implemented with a VueJS frontend web application and a NodeJS backend to handle all asynchronous calls.  Queries are sent concurrently whenever possible.  Source code is withheld to maintain anonymity and will be released in the final publication.







\label{sec:system}


\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 1: Conflict Data}



To demonstrate the value of column augmentation  for insight generation, we present a \DIFdelbegin \DIFdel{use case }\DIFdelend \DIFaddbegin \DIFadd{usage scenario }\DIFaddend in which Auger\xspace is integrated into a typical workflow for the popular visual analytics tool, Tableau.   Suppose a political scientist wants to study the factors that lead to armed conflict.  They download the Armed Conflict Location \& Event Data Project (ACLED) dataset for April 2018, which contains \DIFdelbegin \DIFdel{hundreds of }\DIFdelend \DIFaddbegin \DIFadd{2,279 }\DIFaddend records of armed conflicts across many different countries and \DIFdelbegin \DIFdel{times}\DIFdelend \DIFaddbegin \DIFadd{dates}\DIFaddend \cite{raleigh2010introducing}.  Initially, they load the data into Tableau to analyze relationships between countries\DIFdelbegin \DIFdel{and event types.  They produce a heatmap, seen in Figure~\ref{fig:tableau_use_case}(a), to try to detect patterns, but }\DIFdelend \DIFaddbegin \DIFadd{, event types, and fatalities stemming from armed conflict.  They see no clear pattern that emerges as they }\DIFaddend find it difficult to visually group countries without more data.

\DIFaddbegin 

\DIFaddend They load their dataset into Auger\xspace to look for additional attributes that may be relevant to event types.  When the system is first loaded, the user sees a list of the attributes in the uploaded table in Figure~\ref{fig:system_overview}-A.  They also can see the first five rows of the data below in Figure~\ref{fig:system_overview}-D.  

The political scientist believes \DIFdelbegin \DIFdel{the type of government could be related to the event type}\DIFdelend \DIFaddbegin \DIFadd{there might be economic metrics that effect the types and severity of conflicts}\DIFaddend , so they click the \textit{related attributes} button next to ``Country'' in Figure~\ref{fig:system_overview}-A.  The system returns a list of related attributes found by scraping Wikidata, as seen in Figure~\ref{fig:system_overview}-B (\textbf{T1}).  The user scans through the list of related attributes, looking for information about the \DIFdelbegin \DIFdel{type of government}\DIFdelend \DIFaddbegin \DIFadd{countries' economies}\DIFaddend .  They see that Wikidata holds the \textit{\DIFdelbegin \DIFdel{basic form of government}\DIFdelend \DIFaddbegin \DIFadd{nominal GDP per capita}\DIFaddend } for each country.  
\DIFdelbegin \DIFdel{The tooltip for that attribute, seen in Figure~\ref{fig:system_overview}-C, provides more information to affirm that this holds the right data by showing the user examples of the attribute, like ``unitary state,'' ``federal republic,'' and ``absolute monarchy'' (}\textbf{\DIFdel{T2}
\DIFdel{).  The user }\DIFdelend
\DIFaddbegin \DIFadd{The user }\DIFaddend adds this attribute to the dataset.  While looking for \DIFdelbegin \DIFdel{the form of government}\DIFdelend \DIFaddbegin \DIFadd{economic data}\DIFaddend , the user also notices additional interesting \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend for analysis and joins them for future analysis, including the max \textit{inflation rate}, minimum \textit{life expectancy}, \DIFdelbegin \DIFdel{mean }\DIFdelend \DIFaddbegin \DIFadd{the }\DIFaddend \textit{\DIFdelbegin \DIFdel{nominal GDP per capita}\DIFdelend \DIFaddbegin \DIFadd{type of government}\DIFaddend }, and the mean \textit{Human Development Index} (\textbf{T3}).

The user loads this next iteration of the dataset into Tableau, and generates \DIFdelbegin \DIFdel{the same visualization but with the basic form of government replacing the country name, }\DIFdelend \DIFaddbegin \DIFadd{various visualizations including the event types and fatality numbers with their newly-discovered attributes.  As can be }\DIFaddend seen in Figure~\ref{fig:tableau_use_case}\DIFdelbegin \DIFdel{(b).  While some patterns begin to emerge, the user notes that the basic form of government is too fine-grained for the type of analysis they want to do, since most categories only have one or two countries in them.  They return to 
\DIFdel{to see if there is any higher-level categorization available for that column by searching for attributes off of that attribute (}\textbf{\DIFdel{T4}
\DIFdel{).  They find that Wikidata holds a }\textit{\DIFdel{subclass of}}
\DIFdel{attribute for basic forms of government with examples including ``democracy, '' ``monarchy,'' ``republic,'' and add that attribute using a through join before returning to Tableau to generate the final visualization, seen in Figure~\ref{fig:tableau_use_case}(c).  Now, they can see a pattern in that conflicts concentrate on republics in the region, with democracies and monarchies having interesting patterns in their types of conflicts as well.  This use case }\DIFdelend \DIFaddbegin \DIFadd{, grouping event types by nominal GDP per capita reveals a trend that certain event types, like }\textit{\DIFadd{Remote violence}} \DIFadd{and }\textit{\DIFadd{Battle - No change of territory}}\DIFadd{, appear to have more severe fatalities in countries that have lower nominal GDP per capita.  In contrast, other event types like }\textit{\DIFadd{Battle - Government regains territory}} \DIFadd{and }\textit{\DIFadd{Battle - Non state actor overtakes territory}} \DIFadd{don't seem to be very different between high nominal GDP per capita countries and low ones.  The additional visual analysis enabled by the added attribute, nominal GDP per capita, has led the user to new hypotheses that can then be further explored.  This usage scenario }\DIFaddend demonstrates how the integration of column augmentation into a traditional insight generation pipeline can unlock new analyses in-situ.

\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 2: Modeling Unemployment}

\label{sec:use_cases_ml}
\DIFdelbegin

\DIFdelFL{A modified version of 
\DIFdelFL{used for both Use Case 2 and Task 2 of the user study. Users can build predictive models as they augment their dataset to help them discover attributes that help predict a target variable.  This figure portrays the construction of regression models to predict the rate of unemployment in counties of the United States.  The history of trained models is shown at the bottom of the column view.   By mousing over a model, the user can see the five highest feature importance scores for that model, and the list of attributes used to train that model are highlighted in black.}}
   
\DIFdelend

In our second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate the value of \DIFdelbegin \DIFdel{Auger }\DIFdelend \DIFaddbegin Auger\xspace \DIFaddend for a separate type of visual analytics task - the generation of a predictive model.  Many visual analytics systems have been constructed to enable domain experts to interact with and steer the generation of machine learning models on their data.  Augmenting a dataset with new \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend is one way in which a user can imbue domain knowledge into the modeling process.  They may know that certain attributes can help the prediction while others may mislead.  While it may seem unusual to augment a training set with additional attributes, the additional attributes found on the knowledge graph can likewise be added to any data that the resulting model is asked to predict on at test time when the model is deployed.

\DIFdelbegin \DIFdel{We }\DIFdelend \DIFaddbegin \DIFadd{To demonstrate the use of CAVA in the model building process, we }\DIFaddend modify Auger\xspace to be able to train a predictive model at any point in the data augmentation process\DIFdelbegin \DIFdel{(see Figure~\ref{fig:ml_use_case}). 
When the user instructs 
\DIFdelend \DIFaddbegin \DIFadd{. 
Users can instruct the system }\DIFaddend to build a \DIFdelbegin \DIFdel{model, }\DIFdelend \DIFaddbegin \DIFadd{predictive model with the current version of the data.  }\DIFaddend Auger\xspace splits the data into a training and test set in the ratio of 0.8:0.2 and trains a random forest regressor.  For each model, Auger\xspace reports the \textit{R Squared} score on the training and test sets, as well as the feature importance scores of the five most important features.  \DIFaddbegin \DIFadd{This functionality was designed to be representative the iterative workflow of model builders, and to examine the ways }Auger\xspace \DIFadd{could help in-situ model building.
}\DIFaddend

Consider Andy\xspace is a public policy analyst who seeks to build a regression model on \DIFdelbegin \DIFdel{the unemployment rate dataset \mbox
\cite{unemployment2015} }\hspace{0pt
with $1200$ data samples from the year $2015$. Every row in the data is a county and contains a sparse set of attributes such as }\textit{\DIFdel{County}
\DIFdel{, }\DIFdelend \DIFaddbegin \DIFadd{a U.S. Census Bureau dataset containing data about poverty in different counties around the U.S.\mbox
\cite{poverty2020}}\hspace{0pt
.
The dataset contains $3136$ rows, and $6$ attributes--the dataset index, }\textit{\DIFadd{FIPS}}\DIFadd{, }\DIFaddend \textit{State}, \textit{\DIFdelbegin \DIFdel{Month}\DIFdelend \DIFaddbegin \DIFadd{County}\DIFaddend }, \DIFdelbegin \DIFdel{and the response variable }\textit{\DIFdel{Unemployment Rate}
\DIFdelend \DIFaddbegin \textit{\DIFadd{RUCCode}}\DIFadd{, and the number of residents living in the county under the poverty line}\DIFaddend .
Andy\xspace seeks to augment this data by adding meaningful columns to accurately predict the \textit{\DIFdelbegin \DIFdel{Unemployment Rate}\DIFdelend \DIFaddbegin \DIFadd{Poverty}\DIFaddend }. 
When Andy\xspace first loads the data in Auger\xspace, an initial model is trained with an \textit{R Squared} score of \DIFdelbegin \DIFdel{$0.646$ }\DIFdelend \DIFaddbegin \DIFadd{$0.577$ }\DIFaddend on training and \DIFdelbegin \DIFdel{$0.292$ }\DIFdelend \DIFaddbegin \DIFadd{$-1.801$ }\DIFaddend on test set respectively\DIFaddbegin \DIFadd{; indicating that the model badly overfitted the training data}\DIFaddend . 
Andy\xspace seeks to improve the regression model's performance further by augmenting the base data using Auger\xspace's workflow and visual interface.
Andy\xspace searches for related attributes of the variable \textit{State} from the Column View. In response, Auger\xspace shows a list of related attributes that Andy\xspace may consider to add to the data. 
From these attributes they click on a set of interesting attributes such as \textit{shares border with}, \textit{\DIFdelbegin \DIFdel{head of government}\DIFdelend \DIFaddbegin \DIFadd{life expectancy}\DIFaddend }, \textit{inflation rate}, etc. to see the distribution of values, and its metadata as a text view. 

From these set of attributes Andy\xspace thinks that the attributes \textit{Inflation Rate} and \textit{\DIFaddbegin \DIFadd{nominal }\DIFaddend GDP per capita} are good attributes to predict \DIFdelbegin \DIFdel{the }\DIFdelend \textit{\DIFdelbegin \DIFdel{unemployment rate}\DIFdelend \DIFaddbegin \DIFadd{Poverty}\DIFaddend } and thus decides to add them to the data using the join operation \textit{\DIFdelbegin \DIFdel{Median}\DIFdelend \DIFaddbegin \DIFadd{Mean}\DIFaddend }. They notice that the newly added column is displayed on the Table View. However looking at a few rows of the table Andy\xspace finds quite a few cells that are empty indicating that the joined data may have missing values. 
Nonetheless, Andy\xspace clicks a button to construct a new regression model using the augmented data.
Andy\xspace notices that the \textit{R Squared} score marginally improves (new score: \DIFdelbegin \DIFdel{$0.712$ and $0.301$ }\DIFdelend \DIFaddbegin \DIFadd{$0.586$ and $-1.810$ }\DIFaddend on training and test set respectively). They hover the mouse over the model metric card to see the list of ``Top $5$'' attributes with their weights utilised in the regression model. 
While Andy\xspace expected to see a substantial improvement in the model performance, 
they infer that the marginal improvement is probably due to missing values in the data after the join operation. \looseness=-1

Motivated to improve the model further, Andy\xspace searches for columns that may directly help to predict \DIFdelbegin \DIFdel{the unemployment rate }\DIFdelend \DIFaddbegin \DIFadd{poverty }\DIFaddend per county. Based on prior knowledge, Andy\xspace understands that the \DIFdelbegin \DIFdel{population }\DIFdelend \DIFaddbegin \textit{\DIFadd{population}} \DIFaddend of a state may be directly proportional to \DIFdelbegin \DIFdel{the unemployment rate}\DIFdelend \DIFaddbegin \DIFadd{its poverty}\DIFaddend , and they add that attribute. 
In the process of searching for other relevant columns, Andy\xspace notices the column \textit{Maximum temperature}. Inquisitive to see if a temperature of a state is correlated with its \textit{\DIFdelbegin \DIFdel{unemployment rate}\DIFdelend \DIFaddbegin \DIFadd{Poverty}\DIFaddend } they add it to the table.
After constructing a new regression model, Andy\xspace notices that the \textit{R Squared} score \DIFdelbegin \DIFdel{improved substantially from $0.712$ to $0.855$ and from $0.301$ to $0.338$ }\DIFdelend \DIFaddbegin \DIFadd{changed from $0.586$ to $0.577$ and from $-1.810$ to $0.541$ }\DIFaddend for the training and test dataset respectively. Happy with the progress so far they decide to remove any column that may not be contributing to the prediction task. First\DIFaddbegin \DIFadd{, }\DIFaddend they remove the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{columns }\DIFaddend \textit{\DIFdelbegin \DIFdel{Month}\DIFdelend \DIFaddbegin \DIFadd{RUC Code}\DIFaddend }\DIFaddbegin \DIFadd{, and }\textit{\DIFadd{nominal GDP per capita}} \DIFaddend (by triggering the slider on the Column View) and then triggers Auger\xspace to construct a new regression model. 
As expected Andy\xspace notices that the \textit{R Squared} score did not change. 
\DIFdelbegin \DIFdel{Next to investigate the contribution }\DIFdelend
\DIFaddbegin \DIFadd{Next }Andy\xspace \DIFadd{searches for related attributes }\DIFaddend of the column \textit{\DIFdelbegin \DIFdel{Maximum temperature
\DIFdel{, they remove it from the data.

\DIFdel{notices that the }\textit{\DIFdel{R Squared}}
\DIFdel{score of the new model dropped marginally meaning that the column }\textit{\DIFdel{Maximum temperature}}
\DIFdel{was helpful in improving the model.
They }\DIFdelend \DIFaddbegin \DIFadd{County}}\DIFadd{.
They explore the set of choices shown in the Column View.
They choose to }\DIFaddend add  the column \DIFdelbegin \DIFdel{back to the data}\DIFdelend \DIFaddbegin \textit{\DIFadd{County-population}} \DIFadd{(by median operation), and }\textit{\DIFadd{shares border with}} \DIFadd{(by count operation). }Andy\xspace \DIFadd{constructs a new regression model to see the models' train and test }\textit{\DIFadd{R Squared}} \DIFadd{score improved considerably ($0.807$, and $0.746$ respectively). Content with the improvement, they export the model and the data to continue analysis}\DIFaddend .

In a short time, Andy\xspace has constructed a regression model with substantially better R2 score than the model trained on the base data.  They have also gained insight into which attributes are relevant to their modeling problem, which may help their understanding of the resulting machine learning model.

\DIFdelbegin


\DIFdelend \section{Preliminary User Study}
\label{sec:user_study}
We evaluate Auger\xspace in a preliminary user study. The purpose of the evaluation is to validate Auger\xspace both in terms of its usability and its effectiveness in helping a user improve a machine learning model through data augmentation. 
Specifically, we hypothesize that:

\begin{itemize}
    \item \textbf{H1}: Auger\xspace allows users to accurately join external data given a written description of that data. 
   
   
    \item \textbf{H2}: Auger\xspace is able to help users discover additional data that can improve the predictive quality of machine learning models trained on the dataset.
   
\end{itemize}

We recruited $6$ participants (3 Female, 3 Male), between the age of $23-36$. We required each participant to have at least an elementary knowledge of machine learning and data analysis. Due to COVID-19, we were not able to conduct an in-person study. Instead  we conducted an online study using Bluejeans\footnote{https://www.bluejeans.com/}. The participants interacted with Auger\xspace on their own computer while sharing their screen.
We provided the participants with a url to our system that was hosted on our local machine that is exposed using the Ngrok remote tunneling software\footnote{https://ngrok.com/}. The study took approximately $50-60$ minutes and we compensated the participants with a $\$10$ Amazon gift card.


\subsection{Study Design}
Before the study we asked participants to fill out a background information questionnaire regarding their name, age, gender, machine learning expertise and various use cases in which they use machine learning.
We began the study by showing the participants a tutorial video of Auger\xspace, explaining the workflow, interface GUI elements, and its interaction capabilities that support various join operations. Next we asked the participants to perform three tasks, the first of which was a practice task to ensure that they were 
sufficiently knowledgeable about Auger\xspace to perform the experimental trials. We proceeded to the experimental sessions only when we observed that the participants were confident and able to use Auger\xspace on their own. In the next two tasks we asked the participants: (1) To add a set of specified \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to a given dataset. These \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend can be added to the data using various join operations supported by Auger\xspace. (2) To freely augment the data such that they can improve the performance metric of a machine learning model (metric being a R Squared Score of a regression model). 
We used the Scikit Learn machine learning library \cite{scikit_ml} to construct \textit{Random Forest Regression} models, in the same manner as described in Section~\ref{sec:use_cases_ml}.
We collected the following data from each experimental trial: (1) \textit{Task competition time}, (2) \textit{Task Accuracy}, (3) \textit{Model performance metric} e.g., R Squared Score, and (4) \textit{User ratings} collected using a post-study Likert-scale questionnaire.


\subsection{Datasets}

For the tutorial video and practice task we used \DIFdelbegin \DIFdel{a }\DIFdelend \DIFaddbegin \DIFadd{the same }\DIFaddend U.S. Census Bureau dataset \DIFdelbegin \DIFdel{containing data about poverty in different counties around the U.S.\mbox
\cite{poverty2020}}\hspace{0pt
.
The dataset contains $3136$ rows, and $6$ attributes such as }\textit{\DIFdel{FIPS}
\DIFdel{, }\textit{\DIFdel{State}
\DIFdel{, }\textit{\DIFdel{County}
\DIFdel{, }\textit{\DIFdel{RUCCode}
\DIFdel{, and the number of residents living in the county under the poverty line.
}\DIFdelend \DIFaddbegin \DIFadd{as section~\ref{sec:use_cases_ml}.
}\DIFaddend For the first task in our experimental session we use the IMDB Movies dataset\cite{imdb2020} containing $500$ movies and $28$ attributes such as 
\textit{Director-name}, \textit{Duration}, \textit{Movie-Title}, \textit{Movie Cast Facebook Likes}, etc. The second task in the experimental session which required users to augment data and construct regression models, used the unemployment rate dataset\cite{unemployment2015}. This dataset contained $1200$ rows where each row corresponded to a county's poverty rate measured in a given month. This dataset contained only $5$ \DIFdelbegin \DIFdel{attributes such as }\DIFdelend \DIFaddbegin \DIFadd{attributes--the dataset index, }\DIFaddend \textit{Month}, \textit{State}, \textit{County}, and the \textit{Unemployment-Rate} (dependent variable). 

\subsection{Tasks and procedure}
Participants were first asked to complete a practice task.  
During the practice task, their answers were not recorded and their performance was not included in the analysis described below.  
For the practice task, using the Poverty dataset, we asked the participants to add three \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend : (1) the area of the county, (2) the population of the state, and (3) the earliest inception date of any county that each county shared borders with. 
While the first two \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend could be retrieved using a straight-forward \textit{join by value} operation, the third \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend required the \textit{join through} operation to search for the required \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend that was one ``hop'' away in the knowledge graph. For the first experimental task using the IMDB dataset, participants were given descriptions of four attributes to add: (1) the director's country of citizenship, (2) the number of awards received by the director, (3) the number of cast members in each movie, (4) the sum of the number of awards received by all the producers of each movie.  
Note that attributes 3 and 4 required the participants to \textit{join through} several attributes.  

The final experimental task gave participants $10$ minutes 
to augment the Unemployment rate dataset with as many attributes as they'd like that they believed would aid in building an accurate regression model to predict poverty rate.  
At any point in the 10 minutes, participants could instruct Auger\xspace to build a regression model with the current dataset to give them insight into whether they were improving the predictive modeling process.  
Building a model took about 10-20 seconds, and participants would be shown the change in R Squared Score for each new model, as well as the  weights of top ``5'' \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend used in the model. 
Random Forest regression models were used because their flexibility, speed, and accuracy met the constraints of the experiment.


\subsection{Data Collection}

After all the tasks were completed, participants were asked to fill out a post-study questionnaire (using Google Forms) that included: (1) Likert-scale\DIFaddbegin \DIFadd{\mbox
\cite{likert1932technique} }\hspace{0pt
}\DIFaddend rating questions on their use of the system, (2) a NASA-TLX\cite{hart1988development} questionnaire to measure task difficulty, and (3) Open-ended descriptive questions asking users about the interface design, the workflow, and other responses related to improve the usability of the system\footnote{Pre-experiment and Post-experiment surveys are attached as supplemental material}. 
The Likert-scale questions asked the participants to rate (in a scale of $1-7$, $1$ being ``strongly disagree'') if they found the system: (1) Easy to learn, (2) Intuitive, and (3) Expressive for data augmentation. 
We used the open-ended descriptive feedback to qualitatively evaluate the usability and the interaction design of Auger\xspace for their tasks.
Open-ended prompts included (1) \textit{Describe your thoughts about Auger\xspace}, (2) \textit{Describe things you disliked about the system or the workflow. Elaborate on how you think it could have been improved}, and (3) \textit{Describe your strategy or your process to augment the data for Tasks 1 and 2}\DIFaddbegin \DIFadd{.
}\DIFaddend With the consent of the participants,
each session was \DIFaddbegin \DIFadd{both }\DIFaddend video and audio-recorded. We encouraged the participants to verbalize their thoughts following a think-aloud protocol.
Furthermore, to assess if data augmentation using Auger\xspace led to any change in the regression models performance\DIFaddbegin \DIFadd{, }\DIFaddend we saved: (1) model metric (i.e., R Squared Score), (2) \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend weights, and (3) predicted values from the model. We also saved user mouse-clicks to analyze the set of \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend the user explored to augment the data.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/questionnaire_responses_2.pdf}
    \vspace{-8mm}
    \caption{Participants' responses about the system (Qn1--Qn4, the higher the better) and the overall effort (Qn5--Qn9, the lower the better).}
        \vspace{-3mm}

    \label{fig:survey_ratings}
\end{figure}


\subsection{Result and Analysis}
\DIFdelbegin \DIFdel{We report three types of results: task performance, user satisfaction, and qualitative feedback.
}\DIFdelend

\subsubsection{Task Performance}

To assess \textbf{H1}, we reviewed the number of attributes that participants correctly joined in Task 1.
Five out of six participants properly joined all four expected attributes.
Only one participants missed one attribute when joining the number of awards received by the director.
This indicates that in general participants were able to use Auger\xspace to accurately add new data attributes.

Next, to assess \textbf{H2}, we similarly reviewed the \textit{R Squared} score improvement that participants achieved by augmenting additional data in Task 2.
On average, participants improved the \textit{R Squared} Score by $0.048$ ($\sigma=0.014$), starting with a baseline \textit{R Squared} Score of $0.292$. 
The area and population of the county are two most frequently joined attributes across all the participants, which is reasonable as these two attributes are usually the most influential for predicting unemployment rate. 

As a result of our analysis, we \textbf{accept \DIFdelbegin \DIFdel{both }\DIFdelend H1\DIFdelbegin \DIFdel{and H2}\DIFdelend } because all participants were able to successfully use Auger\xspace to explore and identify the correct attributes for data joining in Task \DIFdelbegin \DIFdel{1, and }\DIFdelend \DIFaddbegin \DIFadd{1.
However, we }\textbf{\DIFadd{neither accept nor reject H2}} \DIFadd{as the study results did not contain clear evidence of users being able }\DIFaddend to improve the predictive quality of a regression model in Task \DIFdelbegin \DIFdel{1. 
}\DIFdelend \DIFaddbegin \DIFadd{2 beyond trivial improvements.
}\DIFaddend 



\subsubsection{User satisfaction}

We use the post-study questionnaire to assess user satisfaction of the data augmentation process in Auger\xspace.
\autoref{fig:survey_ratings} shows the results of the questionnaire responses.
Although the participant size is too small to infer statistical significance using quantitative analysis, we do observe that participants found Auger\xspace easy and intuitive to use based on the likert scale user ratings (\autoref{fig:survey_ratings}, \textbf{Qn1,2}).
Participants were also satisfied with the two main functionalities provided by Auger\xspace, finding relevant attributes from the data and joining additional data (\autoref{fig:survey_ratings}, \textbf{Qn3,4}).
The mean ratings of \textbf{Qn1--4} were all 6 or above.
Furthermore, in analyzing user satisfaction related to the overall process (\autoref{fig:survey_ratings}, \textbf{Qn5--9}), we found that participants were generally satisfied with the process of conducting the tasks, as the mean ratings of \textbf{Qn5--9} are all 2 or lower.


\subsubsection{Qualitative Feedback}

To assess \textbf{H1} and \textbf{H2} from a qualitative perspective, we analysed participants’ descriptive feedback collected from post-study interviews. We also observed participants' workflow in using Auger\xspace from audio and video recordings of their computer.
We report the following three main qualitative user responses from the study:

\vspace{3pt} \noindent \textbf{Intuitive workflow:}
All the participants \DIFdelbegin \DIFdel{fount }\DIFdelend \DIFaddbegin \DIFadd{found }\DIFaddend Auger\xspace's workflow intuitive to find and augment relative data attributes.
The primary justification for Auger\xspace's intuitiveness is that Auger\xspace provides a visual interface that is easy to infer and for users to search and add relevant attributes from the data.
As \textit{P6} noted, ``\textit{I liked the interface of this tool because it was easy to navigate and add/remove columns to the base data}.''
The intuitiveness helped most participants to easily perform the requested tasks, especially Task 1.
For example, \textit{P1} described his experience in doing Task 1 as ``\textit{just looked at the question and implemented it}'' and ``\textit{it was pretty straight forward}.''

\vspace{3pt} \noindent \textbf{Two major strategies in model construction:}
When conducting Task 2, we observed two common strategies used by the participants: (1) searching for relevant attributes based on existing or prior knowledge and (2) exploring all possible combinations of available attributes to find one that results in improvement in model performance.
Some participants relied on their existing domain knowledge for augmenting the data with new attributes.
This was summarized nicely by \textit{P5}, ``\textit{I was actually using my domain knowledge. Thinking which factors can improve unemployment rate predictions, and then tried to select the variables based on the choices I had. This actually helped me improve the model's performance}.''
However, a few participants who may lack prior knowledge about the data, instead explored the dataset and tried out all possible combinations of attributes to find a model that is a significant improvement over the model trained on the base data.
``\textit{I selected columns one by one to see the change of R-squared score. By doing so, I was able to filter the variables that decrease R-square. In the end, I was able to get a relatively high R-squared model}'' (\textit{P6}).
The rest of the participants adopted a combination of both strategies.
For example, \textit{P2} described his strategies as follows, ``\textit{Thought a bit about what could boost performance, then I did some trial and error (manual feed-forward selection of attributes)}.''


\vspace{3pt} \noindent \textbf{Need for more powerful system features for expert users:}
Participants with visual analytics backgrounds expected editable visualizations to represent or encode \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend distribution differently.
``\textit{It would be cool to add additional ways of visualizing attributes}'' (\textit{P2}).
What's more, participants with database backgrounds wanted more details about queries of fetching data to increase flexibility.
``\textit{Maybe consider providing the real query of fetching the dataset to the expert users to give them more clear sense and allow them to change the query}” (\textit{P1}).
\DIFdelbegin \DIFdel{Similarly, participants with more expertise in machine learning sought more control over the model building process.
``}\textit{\DIFdel{May be add hyperparameter tuning for the model building phase, such as L1, L2 norm or change learning algorithm of the model}
\DIFdel{'' (}\textit{\DIFdel{P2}
\DIFdel{).
}\DIFdelend
\DIFdelbegin \DIFdel{All these }\DIFdelend \DIFaddbegin \DIFadd{These }\DIFaddend suggestions are valuable for guiding our further improvement in generalizing Auger\xspace to tackle various problem domains.



\subsection{Limitations}

While our user study results \DIFdelbegin \DIFdel{support both }\DIFdelend \textbf{\DIFaddbegin \DIFadd{support }\DIFaddend H1} \DIFdelbegin \DIFdel{and }\DIFdelend \DIFaddbegin \DIFadd{but }\DIFaddend \textbf{\DIFaddbegin \DIFadd{neither accept nor reject }\DIFaddend H2}, the results should be read in the light of the study limitations.
The number of the participants for our study are limited, which might be a confounding point of the results.
As we conducted the study online, the uncertainty existing in the online setup, such as internet connection, might also confound the results.
To improve the performance of the system and the fluidity of user interaction, we limited the number of related attributes that participants could fetch for each parent attribute.
Although it did help participants with a smoother user experience during the study, participants also complained about the restriction as it limited their performance of tasks, especially Task 2.
Despite the limitations, the user study does help us to better identify the limitations in our system design and improve the system with new features.





\section{Discussion}
\label{sec:discussion}

Through our user study and \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we have demonstrated the efficacy of information foraging using knowledge graphs within visual analytics systems.  The generality of Auger\xspace suggests that data augmentation could \DIFaddbegin \DIFadd{be }\DIFaddend added into traditional visual analytics system workflows to improve the outcome of any embedded task.  Many fruitful avenues of research arise when considering how to apply information foraging to the full space of \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend which are served by visual analytics systems.

\subsection{Mental Models of Data Augmentation}

Knowledge graphs can be difficult to reason about, especially when they are used to find data corresponding to an entire column of a dataset.  In Auger\xspace, we address this by showing the user previews of the joined dataset, as well as visualizations of the distribution of joined data and an example of the portion of the knowledge graph that is used to construct a single value (see Figure~\ref{fig:through_join}).  This approach was based on our conversations with designers of visual analytics applications for building predictive models.  A better understanding of the user's mental model of the knowledge graph and the joining process is needed to generalize this approach to other \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend .  We generally aimed to hide the complexity of the underlying knowledge graph, but it may be the case that more direct exploration of the knowledge graph is helpful for some cases.

\DIFaddbegin \subsection{\DIFadd{Sparsity in Knowledge Graphs}}

\label{sec:sparsity}

\DIFadd{In this work, we rely on the high quality of Wikidata to find attributes that have sufficient support for the user's dataset.  But in many cases, only 10-15\% of the rows of a dataset will have joinable data for a given attribute.  Because knowledge graphs do not hold schematic data, they can suffer from data sparsity issues.  We feel that the experiments in this work show that there are many popular data types, including geographic entities like countries or states, that have sufficient robust data to discover attributes with full support on a user's dataset.  In situations where the attribute of interest is only present for a small percent of rows of the dataset, we suggest two possible solutions.  First, the additional data can still be useful as additional information injected into the dataset, and using imputation or setting reasonable default values (using a tool such as the }\textit{\DIFadd{data tamer}} \DIFadd{~\mbox
\cite{Stonebraker2013} }\hspace{0pt
can make that information usable by downstream analyses.  Second, a user can explore the reduced dataset in which the sparse attribute is present, and if it results in promising analysis, they can use }Auger\xspace \DIFadd{to search for other more complete attributes that might be correlated with that sparse attribute, based on their domain expertise.  Lastly, we would like to point out that knowledge graphs are not read-only structures, and }Auger\xspace \DIFadd{may help point out where it would be valuable for an organization to invest in recording more data.
}

\DIFaddend \subsection{Design Space for Interactions with Knowledge Graphs}

In Auger\xspace, the channel between the user and the knowledge graph is limited to the two subroutines shown in section~\ref{sec:technique}, which allow the user to see lists of related attributes and then construct queries given that information.  However, there is more potential to improve the user's control over the process and address edge cases by expanding the set of interactions between user and knowledge graph.  Automated processes in Auger\xspace could be replaced by collaborations between user and system.  

For example, ambiguities in the entity resolution used to retrieve a list of related attributes can be solved by user interaction.  A list of countries could refer to the governmental entities they describe, or they could refer to the national soccer teams participating in the World Cup; there will always exist cases where disambiguating the type of \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend will require domain expertise from a user.  

The user could also benefit from more fine-grained control over the process of building aggregation queries.  Users may want to join timestamped data, which would necessitate some specification from the user of how to parse and interpret temporal columns.  Spatial data offers an additional potential, as the user might want to use geographical data in their dataset to search for the closest weather station or other geographically tagged entity.  There are many types of queries that might necessitate different user interactions than have been used in previous visual analytics systems.


\DIFdelbegin \subsection{\DIFdel{Scalability}}
\addtocounter{subsection}{-1
\DIFdelend

\DIFdelbegin \DIFdel{In order for 
\DIFdel{to operate at the speed necessary for exploration, it makes use of only samples of the data, rather than the entire dataset .  When gathering the list of related attributes for a column, it samples only a subset of values from that column.  Those related attributes are then shown with estimations of their distribution as well as what percent of rows have a joinable value.  While we use a sampling-based approach, there is plenty of opportunity to develop methods that provide better estimates or allow for joining to more data in less time.  
The database community is working on tools for query optimization over knowledge graphs\mbox
\cite{Mohamed2020}}\hspace{0pt
, which will help.  But specific indices, query optimizations, and data structures should be explored by the visual analytics community, the same way that many advancements in querying tabular data have been made to allow for faster data exploration\mbox
\cite{lins2013nanocubes, wang2018neuralcubes, pahins2016hashedcubes, liu2013immens, tao2019kyrix}}\hspace{0pt
.
}\DIFdelend






\section{Conclusion}
\label{sec:conclusion}
In this work, we presented Auger\xspace, a visual analytics system for explorative information foraging using knowledge graphs.  Knowledge graphs offer a wealth of information on a broad array of topics that could be used to improve the outcome of embedded tasks of visual analytics systems.  In our \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we demonstrated that the data gathered through Auger\xspace could result in better predictive models and better insight generation on two datasets.  And in our experiment, we showed that Auger\xspace is simple to learn and use, as all six of our participants were able to effectively explore and join data on multiple datasets within a 60 minute session.  


\DIFdelbegin

\DIFdelend \DIFaddbegin \acknowledgments{
We thank our collaborators in DARPA's Data Driven Discovery of Models (d3m) program, as well as the reviewers for their helpful feedback.  This work was supported by National Science Foundation grants IIS-1452977, OAC-1940175, OAC-1939945, DGE-1855886, and 1841349, as well as DARPA grant FA8750-17-2-0107.
}
\DIFaddend
\bibliographystyle{abbrv-doi}


\subsection{New Intro}



Over 20 years ago, Pirolli and Card coined the term ``information foraging'' to denote the process of seeking and gathering information to apply towards a task\cite{pirolli1999information, pirolli2005sensemaking}.  While they focused on foraging for additional entities or rows of a dataset, foraging for additional data attributes \DIFaddbegin \DIFadd{or columns }\DIFaddend can unlock new analytical capabilities.  For example, if a dataset contains a list of countries, adding the population of those countries as \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend enables per-capita analyses. 

Traditionally, crafting new attributes and augmenting a dataset with them is done prior to any visual analysis.  Incorporating this process into visual analytics workflows can benefit both the augmentation process as well as the underlying tasks of the system in several ways.  First, the need for an additional \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend may only arise based on insights generated during analysis; if data augmentation is embedded in the system, users can toggle back and forth between analysis and augmentation.  Second, the process of discovering data and crafting a new attribute is an analytical task in its own right, and typically consists of complex querying that can be abstracted away by a visual interface.  Lastly, user-driven curation of the \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of the dataset can serve as an additional medium of communication between user and system: the user can communicate domain knowledge by adding new \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend , and can likewise learn from the attributes that are discovered by the system.

However, there are many complications in integrating data augmentation into visual analytics systems.  Finding external data and matching it to the entites in the user's dataset is nontrivial.  Once connected to external data, it can be difficult to determine which data is relevant or useful to the user's analysis.  Some attributes may require the aggregation of multiple pieces of external data and can require joining through several intermediate entities.  These hurdles should be abstracted away so that the user is not required to be a database expert.  

Large scale information repositories provide the potential for interactive data augmentation because they put the potential data at the user's fingertips. However, in order to use such repositories a number of challenges must be addressed. If the repository is in the form of a data \DIFdelbegin \DIFdel{pool }\DIFdelend \DIFaddbegin \DIFadd{lake }\DIFaddend (a large collection of tables, e.g. WikiTables\cite{bhagavatula2013methodswikitable}), it can be difficult to determine which tables are relevant and joinable to the user's dataset. Tabular data can often be fragmented as well: for example, if a user is foraging for data on water usage across the United States, each municipality might be responsible for publishing its own data, and the formats may not align, resulting in sparse, error-prone joins.  Joining together multiple tables can also result in ambiguities over the type of join (left, right, outer, inner) that are difficult to resolve, and limit the types of attributes that can be constructed without additional data munging.  In addition, plenty of external data may be irrelevant or even harmful to the analysis tasks at hand.  
While there have been great strides in addressing some of the technical issues in using \DIFdelbegin \DIFdel{Data Lakes }\DIFdelend \DIFaddbegin \DIFadd{data lakes }\DIFaddend (see section~\ref{sec:related_work:data_augmentation}), little work has been done to allow a user to explore potential attributes and construct new ones.  Knowledge graphs offer an alternative to data lakes \DIFdelbegin \DIFdel{, }\DIFdelend because they simplify entity matching and joining ambiguities due to their graph format.  They bring different challenges, however, in scalability and complexity, and there is still a dearth of tools available to facilitate foraging over knowledge graphs.

In this paper, we present Auger\xspace, a visual analytics system for \DIFdelbegin \DIFdel{augmenting datasets with carefully crafted columns.  These columns are retrieved }\DIFdelend \DIFaddbegin \DIFadd{Columnar data Augmentation through Visual Analytics.  Carefully crafted attributes are synthesized }\DIFaddend by running queries over knowledge graphs, as illustrated in Figure~\ref{fig:teaser}\DIFaddbegin \DIFadd{, before being added to a dataset as additional columns for downstream analysis}\DIFaddend .  First, each row of a dataset is mapped to an entity in a knowledge graph. Then, to help the user identify potential information of interest, Auger\xspace explores the local graph neighborhood about entities in the dataset to determine commonly held attributes.  Using visualizations of data quality, distribution, and the local topology of the knowledge graph, Auger\xspace guides the user through crafting additional \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend of data without any programming or explicit querying.  Users can express complicated operations over the knowledge graph by interacting with these visualizations. In sum, Auger\xspace abstracts away the complexity of searching, retrieving, and joining data so that the user can focus on the exploratory task of determining which attributes are relevant to their analysis.



Auger\xspace shows the promise of integrating column augmentation as a foraging process within a VA system.  \DIFaddbegin \DIFadd{The design process presented in this work in section~\ref{sec:system} can be useful for future visual analytics tools to adopt columnar data augmentation as part of their workflow.  }\DIFaddend We claim that a resulting VA system can make the exploration and discovery of additional attributes interactive, and let users construct complex \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend without programming.  Users are able to discover new attributes as well as create attributes they had in mind.  The construction of the augmented dataset can improve the outcome of downstream analytical tasks, such as insight generation,  predictive modeling, or any other number of tasks that rely on the presence of a robust dataset.  

We offer evidence of these claims by describing two \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend for insight generation and predictive modeling.  In the first \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate how a domain scientist's analysis can be enriched through cycles of in-situ column augmentation and analysis.  And in the second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we show how a user can craft attributes that result in significantly more accurate machine learning models.  We also conduct a preliminary user study on two datasets to assess the usability of Auger\xspace.  The results confirm that users are able to both discover and craft relevant attributes easily and quickly. 


























In this work, we present the following contributions:
\begin{itemize}[topsep=3pt, itemsep=0pt,parsep=3pt]
    \item We present a visual analytics system, Auger\xspace, for exploratory data augmentation using knowledge graphs.  \DIFaddbegin \DIFadd{We also describe the design process of using visualization as a medium for query construction and knowledge graph exploration. 
    }\DIFaddend \item We provide \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend of Auger\xspace being applied to both insight generation and predictive modeling to demonstrate the generality of our approach.    
    \item We conduct a \DIFaddbegin \DIFadd{preliminary }\DIFaddend user study to assess the \DIFdelbegin \DIFdel{ability }\DIFdelend \DIFaddbegin \DIFadd{usability }\DIFaddend of our system \DIFdelbegin \DIFdel{to join both }\DIFdelend \DIFaddbegin \DIFadd{in joining }\DIFaddend semantically meaningful external data \DIFdelbegin \DIFdel{as well as data that improves a predictive model trained on the dataset}\DIFdelend \DIFaddbegin \DIFadd{across two different tasks, offering validation of our design}\DIFaddend .
\end{itemize}



\DIFaddbegin \vspace{-3mm}
\DIFaddend 

\section{Background and Related Work}\label{sec:related_work}

\subsection{Knowledge Graphs}
\DIFaddbegin \DIFadd{Our system uses knowledge graphs as its knowledge representation.  }\DIFaddend Relational databases are one of the most popular formats to store data because they can be implemented and queried effectively and efficiently.
However, recently knowledge graphs have gained popularity for storing very large datasets because their entity-based model is conducive to how humans think about data.
Knowledge graphs represent data entities as nodes in a graph and the edges between those nodes are relationships between those entities.
The term \textit{knowledge graph} has been loosely used to describe collections of information.
Several different definitions have been offered in recent years\DIFaddbegin \DIFadd{~\mbox
\cite{hogan2020knowledge}}\hspace{0pt
}\DIFaddend .
In practice the term has been used interchangeably with \textit{knowledge base} or \textit{ontology}\cite{ehrlinger2016towards}.
Structuring entities in the form of an ontology is also used to organize conceptual spaces, for example for the evaluation of visual analytics systems\cite{chen2019ontological} and visualizations that support machine learning tasks\cite{sacha2018vis4ml}.

The current popularity of knowledge graphs to store information can be traced back to 2012, when Google introduced its \emph{Knowledge Graph}.
Google's Knowledge Graph is a repository that retrieves facts about entities in search terms on their search results pages\cite{googleknowledgegraph2012, Dong2014}, and is used to power the Google Assistant\cite{googleAssistant} and Google Home voice queries\cite{googleHome}. Recent work has demonstrated that the approach can generalize to other artificial intelligence tasks such as image captioning and conversational agents\cite{huang2019knowledge,zheng2018question,hixon2015learning}.

Since 2012, Wikipedia and other Wikimedia projects have collected information into the \textit{Wikidata} knowledge graph.
It contains broad information about tens of millions of entities.
Other large public repositories exist, from DBPedia\cite{auer2007dbpedia}, a collection of data resources, to the domain-specific universal protein knowledgebase \textit{UniProt}\cite{uniprot2017uniprot} and the linguistic resource \textit{WordNet}\cite{Miller1995}.
Many of these graphs are part of the Linked Open Data Cloud\cite{linkedopendata2020}, a knowledge graph that contains information about other knowledge graphs.
All these graphs contain a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

\DIFdelbegin \DIFdel{A lot of information is available in machine-readable form in knowledge graphs.
With }\DIFdelend \DIFaddbegin \DIFadd{Knowledge graphs have been become popular in many research areas such as information retrieval, Natural Language Processing (NLP), and text analysis. For example, a number of approaches (e.g.,~\mbox
\cite{Dalton2014, liu2015, Rao2013, Raviv2016}}\hspace{0pt
) and tools (e.g.,~\mbox
\cite{Ferragina2010, Gabrilovich2013}}\hspace{0pt
) focus on text-centric information retrieval using entity links in underlying knowledge graphs. Examples of applications include the discovery of emerging entities~\mbox
\cite{Hoffart2014}}\hspace{0pt
, the structuring of contents into topics~\mbox
\cite{Balasubramanian2010}}\hspace{0pt
, extracting relations~\mbox
\cite{Mintz2009}}\hspace{0pt
, semantic search~\mbox
\cite{Berant2013, Krishnan2018, Nanni2018}}\hspace{0pt
, or predicting the missing relations between entities~\mbox
\cite{Dettmers2018, Yang2015}}\hspace{0pt
. Researchers in NLP and text analysis fields have also utilized knowledge graphs from different perspectives, e.g., for embedding entities and words into a continuous vector space~\mbox
\cite{Wang2014}}\hspace{0pt
, for extracting relation facts form text~\mbox
\cite{Riedel2010, Hoffmann2011}}\hspace{0pt
, for the generation of questions and answers pairs automatically~\mbox
\cite{Reddy2017}}\hspace{0pt
, and for parsing and interpretation of user natural language semantically~\mbox
\cite{Hakkani2014}}\hspace{0pt
.
}

\DIFadd{In our system, }\DIFaddend Auger\xspace\DIFaddbegin \DIFadd{, }\DIFaddend users can tap \DIFdelbegin \DIFdel{these }\DIFdelend knowledge repositories as a data source to augment a dataset \DIFaddbegin \DIFadd{through interactive visual support}\DIFaddend . Knowledge graphs have the advantage of providing clean, curated data which means that the need for data cleaning and matching is greatly reduced.
In addition, the entity-focused way in which knowledge graphs store information can make it easier to think about the relationship of data objects and thus helps users guide the augmentation process. 



\subsection{Data Augmentation}
\label{sec:related_work:data_augmentation}
\DIFaddbegin \DIFadd{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new attributes (columns) for the objects.
For example, in image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.
}

\DIFadd{Approaches to add columns to a dataset typically vary in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new attributes from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}

\DIFaddend Automated augmentation of datasets is of interest to the database and the data science communities.
In the database community, the goal of data augmentation can manifest in many ways. 
For example, before augmentation can take place, the first challenge is to find datasets that are suitable for joining. 
In this scenario, sometimes referred to as a ``data lake''\cite{Miller2018}, the data joining system assumes that there is a finite number of candidate datasets, and the task is to identify which of them can be joined with a user's base dataset.
Systems such as Google Goods\cite{halevy2016goods}, Infogather\cite{Yakout2012}, Octopus\cite{cafarella2009data}\DIFaddbegin \DIFadd{, }\DIFaddend Aurum\cite{fernandez2018aurum}, and work by Sarma et al.\cite{das2012finding} examine the attributes of the candidate datasets and learn the relationship between those attributes and the attributes of the input data.

\DIFdelbegin \DIFdel{Another type of data augmentation appears in the form of }\DIFdelend \DIFaddbegin \DIFadd{One challenge of automated data augmentation that is an active area of research is }\DIFaddend entity matching. 
Entity matching refers to finding the same entity in different datasets, often using machine learning techniques. Once identified, the entity can be removed (for deduplication) or merged (for augmentation). Examples of entity matching systems include Magellan\cite{konda2016magellan}, Nadeef\cite{dallachiesa2013nadeef}, Autojoin\cite{zhu2017auto}, and work by Mudgal et al.\cite{mudgal2018deep}.

What the data joining and entity matching approaches have in common is that the systems assume little knowledge about the data. The challenge is therefore to identify the commonalities between the datasets (e.g. schema matching) to determine if the datasets or the entities within are related and therefore joinable.
In Auger\xspace, we take a different approach.
Instead of assuming little to no knowledge about the candidate data, we use a knowledge graph as the source of ``candidate data.'' 
Although knowledge graphs can be loosely structured, they contain well-defined relations between entities. 
This property of knowledge graphs allows Auger\xspace find relevant data for augmentation with certainty and ease.





\DIFdelbegin \DIFdel{In the machine learning community, the goal of data augmentation is to expand a training dataset so that as much of the phenomenon being modeled is present as possible.
This is done in two ways: adding objects (rows) to the training data, and adding new features (columns) for the objects.
For image datasets, adding new objects in the form of slightly modified versions of the training objects (i.e. rotating, adding noise, cropping, modifying color) can improve a machine learning model's sensitivity to noise in the data\mbox
\cite{krizhevsky2012imagenet, Shorten2019, perez2017effectiveness, Jung2020, Park2020}}\hspace{0pt
.


\DIFdelend
\DIFdelbegin \DIFdel{Approaches to add columns to a dataset typically differ in where that data comes from, and whether that data is used in training the model or in model inference.
Feature engineering\mbox
\cite{kanter2015deep} }\hspace{0pt
is a common method to derive new features from existing columns by applying operations to them, such as the difference of two columns.  However, it is limited in that it can only express data that is in the scope of existing attributes.
Knowledge graphs have recently been used to incorporate world knowledge into machine learning models~\mbox
\cite{Song2017} }\hspace{0pt
for a range of models, including text processing\mbox
\cite{Annervaz2018}}\hspace{0pt
, image classification\mbox
\cite{Marino2017}}\hspace{0pt
, and machine translation\mbox
\cite{Moussallem2019}}\hspace{0pt
, but most work incorporates knowledge graphs into the last step of the machine learning pipeline, inference, to gather facts, rather than augmenting an entire dataset.
The Python library RDFFrames\mbox
\cite{Mohamed2020} }\hspace{0pt
helps extract data from knowledge graphs to improve machine learning training.
It allows practitioners to effectively express queries to knowledge graphs and execute them efficiently.  However, it requires extensive data science and programming experience to use.
}\DIFdelend 


While automated approaches can discover joinable attributes for a given dataset, they lack semantic knowledge of expert users.
This may result in a large number of attributes that are irrelevant to the analysis problem being added to the dataset, hindering the users in further analyzing and gaining insights from the dataset.
\DIFdelbegin \DIFdel{In addition, in common one-to-many relationships }\DIFdelend \DIFaddbegin \DIFadd{To make use of their semantic knowledge to choose relevant attributes, users may benefit from an exploratory process facilitated by visual analytics.  Beyond the selection of relevant attributes, users may benefit from visual explanations of complex queries.  For example, in transitive relationships that go through multiple entities}\DIFaddend , joins require \DIFdelbegin \DIFdel{some user guidance }\DIFdelend \DIFaddbegin \DIFadd{multiple operations to be specified }\DIFaddend for aggregating a collection into a single value.
Auger\xspace overcomes these issues by involving a user in the process.  Attributes are only added to the dataset if the user decides that they are helpful for their analysis.  

\begin{comment}
Consider a geospatial dataset, perhaps of temperature readings, with a column representing the country in which the reading took place.  In a broad data repository such as Wikidata, over 1200 attributes are potentially joinable to countries in just a single hop on the knowledge graph.  Joining irrelevant attributes into a dataset can complicate the downstream analytical tasks because of the curse of dimensionality.  

Even if an automated approach were to only select a manageable number of attributes to join, there may be semantic constraints that render certain attributes unusable that only a user would know about.  For example, consider an analyst who is studying housing prices in the United States.  If their task is to build a model for mortgage approvals, there may be certain attributes such as race that are considered protected characteristics that legally cannot be included in the analysis.  On the other hand, they may intentionally want to include these attributes so that they can do a post-hoc fairness analysis to see if those attributes can be predicted from the rest of the dataset~\cite{chouldechova2018frontiers}.  Using automated tools can be dangerous in many analysis environments, and we posit that a human in the loop tool like Auger\xspace mitigates those dangers. It is possible that the performance of Auger\xspace could be improved by using the data discovery system of Aurum rather than Wikidata or making use of the efficient queries from RDFFrames.  In addition, in one-to-many relationships, joins require some user guidance for aggregating a collection into a single value.  For more details on how this is handled in Auger\xspace, see section~\ref{sec:system}.
\end{comment}



\subsection{\DIFdelbegin \DIFdel{Interactive }\DIFdelend Visual \DIFdelbegin \DIFdel{Data Curation}\DIFdelend \DIFaddbegin \DIFadd{Analytics Approaches}\DIFaddend }
\DIFaddbegin 

\DIFaddend Curating, improving, and augmenting an existing dataset \DIFdelbegin \DIFdel{is typically }\DIFdelend \DIFaddbegin \DIFadd{has been a popular research topic in visual analytics.  This can be }\DIFaddend done to aid interpretation and sensemaking\cite{cramer2017impact} during analysis.  \DIFdelbegin \DIFdel{Sometimes referred to as ``data blending'', many commercial visualization systems, such as Tableau\mbox
\cite{Tableau2020}}\hspace{0pt
, Alteryx\mbox
\cite{Alteryx2020}}\hspace{0pt
, and D:Swarm\mbox
\cite{DSwarm2020} }\hspace{0pt
support it by allowing users to join data tables.
The Google Data Studio\mbox
\cite{GoogleDataStudio2020} }\hspace{0pt
and OpenRefine\mbox
\cite{OpenRefine2020}}\hspace{0pt
, both based on the Webtables project\mbox
\cite{Cafarella2008}}\hspace{0pt
, also support querying web APIs
and integrating data from them into a table.
In contrast to our approach, none of these tools supports users with discovering new data for augmentation in a systematic way.
Users have to find additional data on their own before they can join it to the base dataset.


\DIFdelend Other visual interactive approaches allow users to \DIFdelbegin \DIFdel{use and integrate }\DIFdelend \DIFaddbegin \DIFadd{query and integrate query }\DIFaddend data from heterogeneous web \DIFaddbegin \DIFadd{or local }\DIFaddend sources.
This includes helping users extract information from textual web sources\cite{Hoeber2017}, query knowledge graphs\cite{Hoefler2014}, \DIFaddbegin \DIFadd{query large metadata-rich heterogenous data stores~\mbox
\cite{Smith2006}}\hspace{0pt
, identify relations at attribute levels in mixed data sets~\mbox
\cite{Jurgen2014}}\hspace{0pt
, analyse associated categories in large categorial data tables~\mbox
\cite{ Alsallakh2012}}\hspace{0pt
, }\DIFaddend or automatically create visual representations of information stored in knowledge graphs. VAiRoma helps users extract and combine information from Wikipedia articles to create visualizations that provide insight into historical events\cite{Cho2015}.
Vispedia\cite{Chan2008} lets users interactively collect and integrate data from Wikipedia tables to create visualizations and answer analysis questions.
\DIFaddbegin \DIFadd{Atlasify enables users to relate the query concept, corresponds to a Wikipedia article, to spatial entity in the underlying reference system, e.g., countries, political figure~\mbox
\cite{Hecht2012}}\hspace{0pt
.
}\DIFaddend All of these approaches help users access information stored in different forms, but compared to Auger\xspace do not support the process of augmenting an existing dataset with additional data.



\DIFdelbegin \DIFdel{Tools for data wrangling, such as Wrangler, help users to interactively identify data quality problems, and fix them to improve their datasets\mbox
\cite{Kandel2011wrangler}}\hspace{0pt
.
Data wrangling is an important step towards integrating data from diverse sources\mbox
\cite{Stonebraker2013}}\hspace{0pt
.  In 
\DIFdel{, we leave issues of data wrangling to future research, and focus on the pressing need for the discovery of relevant attributes and their extraction from knowledge graphs.  The resulting datasets could then be fed into a tool like Wrangler.
}\DIFdelend

\DIFdelbegin \DIFdel{Data improvement and curation tasks are also supported as part of creating and analyzing machine learning models.
Some approaches directly allow users to add and modify data by providing labels or corrections for train or test data\mbox
\cite{Heimerl2012, Legg2019, mansoor2019delfi}}\hspace{0pt
.
Other systems support identifying mislabeled\mbox
\cite{Ren2016, Alsallakh2017, strobelt2018s} }\hspace{0pt
or generally low quality instances\mbox
\cite{Muehlbacher2013, Gleicher2020} }\hspace{0pt
for a range of different model types.
In contrast, 
\DIFdel{does not help identifying single instances to modify, but helps users adding relevant high-quality data columns to the entire dataset.
}\DIFdelend

\DIFaddbegin 


\DIFaddend \begin{comment}
ML support in fixing data and labels:
- Helping users with correcting recorded and user labeled data~\cite{mansoor2019delfi}
- Boxer is relevant here, too (can help to find mislabeled instances) \cite{Gleicher2020}
- Also Heimerl~et~al.~\cite{Heimerl2012} active learning is a way to effectively label datasets (adding a new column to data items). (also AL: \cite{Legg2019})
- Squares\cite{Ren2016} helps users find mislabled instances in ML (classification) models, but does not support relabeling them directly. One use case shows how mislabeled instances ca be identified. Alsallakh~\cite{Alsallakh2017} claims that as well (but no use case).
Lot of work on finding mislabeled data.  Off the top of my head, Seq2Seq vis has debugging in the title. \cite{strobelt2018s}.
- M\"uhlbacher and Piringer~\cite{Muehlbacher2013} allow users to filter low quality instances to increase model quality.
\end{comment}





















\section{Knowledge Graphs}

\label{sec:knowledge_graphs}



\begin{comment}
Knowledge graphs are a generic data format in which entities are represented as nodes on a graph and the edges between those nodes represent relationships between those entities. They are used because of their flexibility and the ease of reading and writing information about entities.  Much of the current concept of knowledge graphs can be traced back to 2012, when Google introduced its Knowledge Graph, an information repository which retrieves facts about entities in search terms on their search results pages~\cite{googleknowledgegraph2012}.  Since then, Wikipedia and other Wikimedia projects have collected information into \textit{Wikidata}, a knowledge graph with broad information about tens of millions of entities.  Other large public repositories exist ranging from the collection of datasets in \textit{DBpedia}~\cite{auer2007dbpedia} to the universal protein knowledgebase \textit{UniProt}~\cite{uniprot2017uniprot}, and in fact, many of these knowledge graphs are linked themselves in the Linked Open Data Cloud~\cite{linkedopendata2020}.  These knowledge graphs represent a tremendous wealth of structured data that can be accessed programmatically with little to no data munging.

The term \textit{knowledge graph} has been loosely used to describe collections of information.   Several different definitions have been offered in recent years, and in practice the term has been used interchangably with \textit{knowledge base} or \textit{ontology}~\cite{ehrlinger2016towards}.
\end{comment}

We define our usage of the term, knowledge graph, in this section, and describe the sufficient characteristics of a knowledge graph that we assume for our system. We assume that knowledge graphs contain entities as nodes and express attributes of those entities as edges, connecting the source entity to either another entity (to express a type of relationship) or to a literal (such as a string, number or datetime).  The number of edges connected to a node can be a proxy for the number of attributes that are held by that entity.  The flexibility of this structure allows for complex relationships to be expressed between entities, including one-to-one (i.e. a country has one head of state), one-to-many (a country is composed of multiple municipalities), and many-to-many (countries share borders with other countries non-exclusively).


Auger\xspace requires that the connected knowledge graph has a data retrieval endpoint that can respond to simple queries about entities and their neighbors.  The connected graph must also have some sort of service to map from the values in the dataset uploaded to Auger\xspace to the entities on that knowledge graph.  For example, if a dataset with U.S. states is uploaded, there must be an existing service that maps from ``New York'' or ``NY'' to the entity in the knowledge graph corresponding to that state.  These two requirements allow Auger\xspace to connect a user's data to the knowledge graph and to retrieve relevant data for the user to forage from.


In the experiments in this paper, we connect Auger\xspace to Wikidata.  It meets both requirements listed above: it responds to the SPARQL query language so gathering neighborhood data about entities is simple, and it has a service \verb wbsearchentities \hspace{1mm}to map strings to entities on the graph.  But it also has some extra features that we take advantage of.  Wikidata contains a broad set of information making it easy to find related attributes for many different kinds of datasets. It also has additional metadata about the data in its graph including data-type information and human readable descriptions and labels for each node and edge in the graph.  Lastly, due to its popularity, there is a large amount of documentation and guidance on constructing complex queries.  

\DIFaddbegin \DIFadd{In future work, we hope to allow users to connect }Auger\xspace \DIFadd{to multiple knowledge graphs.  To limit scope in this work, we focused only on Wikidata because of its high standards for data quality.  In addition, it contains general knowledge that made it an applicable information repository for a number of different datasets and usage scenarios.  This also made it easier to recruit users for the study described in section~\ref{sec:user_study}.  While different knowledge graphs bring different challenges, particularly around data sparsity (see the discussion in section~\ref{sec:sparsity}), Wikidata provided a broad enough testbed to inform the design goals and interactions supported by }Auger\xspace\DIFadd{.
}

\DIFaddend


\section{Tasks and Goals}



Auger\xspace is developed as part of the DARPA Data-Driven Discovery \DIFdelbegin \DIFdel{and Modeling }\DIFdelend \DIFaddbegin \DIFadd{of Models }\DIFaddend (D3M) program whose goal is to develop software infrastructure and algorithms to make automated machine learning accessible to general data scientists. Inspired by the observation that both the use of exploratory visualization and the use of advanced machine learning are fundamentally limited by the input data, \DIFdelbegin \DIFdel{Auger }\DIFdelend \DIFaddbegin Auger\xspace \DIFaddend was developed to help a data scientist craft better predictive models by foraging for additional \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to add to their dataset.

We conducted interview sessions with four teams within the DARPA D3M program developing applications for data exploration and predictive modeling. The goal of the interview sessions was to better understand how a tool like Auger\xspace can help the user to perform data augmentation for the purpose of improving machine learning model performance and accuracy.
Each of the interviewed teams was shown an early implementation of Auger\xspace that was able to search for related attributes and return a list of them, but without any visualizations. The participants were then asked about what additional \DIFaddbegin \DIFadd{system }\DIFaddend features and interactions would facilitate the discovery of the types of data that they were interested in for their applications.  
\begin{figure*}[thb]
\centering
\includegraphics[width=0.92\textwidth]{figures/full_system_light_acled_labels.png}
\caption{The \DIFdelbeginFL \DIFdelFL{full system }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{user interface of }Auger\xspace\DIFaddFL{, }\DIFaddendFL showing an analysis session on the ACLED dataset.  \textbf{A} shows the initial list of attributes in the table as well as augmented attributes, like the maximum inflation rate recorded for each country.  \DIFdelbeginFL \textbf{\DIFdelFL{B}}
\DIFdelFL{shows the }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{The }\DIFaddendFL list of attributes related to the \DIFdelbeginFL \DIFdelFL{Country }\DIFdelendFL \DIFaddbeginFL \textit{\DIFaddFL{Country}} \DIFaddendFL column \DIFdelbeginFL \DIFdelFL{as discovered from Wikidata}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{is shown in  }\textbf{\DIFaddFL{B}}\DIFaddendFL .  \DIFdelbeginFL \DIFdelFL{The }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Each related }\DIFaddendFL attribute \DIFdelbeginFL \DIFdelFL{of the returned attribute }\DIFdelendFL is \DIFdelbeginFL \DIFdelFL{encoded in }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{adorned with }\DIFaddendFL a \DIFdelbeginFL \DIFdelFL{symbol, a color, and a label on the left side of each row.  The }\DIFdelendFL donut chart \DIFdelbeginFL \DIFdelFL{in each row shows }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{showing }\DIFaddendFL the estimated \DIFdelbeginFL \DIFdelFL{amount of rows of the dataset that will have data available in the connected knowledge graph}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{join quality}\DIFaddendFL .   \DIFdelbeginFL \DIFdelFL{When a row in either of the two columns is clicked, additional metadata }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Additional details }\DIFaddendFL about \DIFdelbeginFL \DIFdelFL{that }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{any }\DIFaddendFL attribute is \DIFdelbeginFL \DIFdelFL{shown }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{available on demand }\DIFaddendFL in a popup, as seen in \textbf{C}.  As attributes are added to the dataset \DIFaddbeginFL \DIFaddFL{during the analysis session}\DIFaddendFL , a preview of the table is updated \DIFdelbeginFL \DIFdelFL{in the raw data table }\DIFdelendFL as seen in \textbf{D}.}

    \vspace{-3mm}

    \label{fig:system_overview}
\end{figure*}


\DIFaddbegin 

\subsection{\DIFadd{Task Analysis}}

\DIFaddend We distilled a list of tasks from the interviews that would enable a user to meaningfully augment their dataset with additional attributes.

\begin{enumerate}[label=\textbf{T\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{View a list of joinable attributes for any existing attribute in the dataset.}  Revealing the list of potential attributes that can be joined with a particular column in the dataset helps the user determine what external data is available for augmentation.  It is a key aspect of the exploration process in information foraging because it may reveal data that the user wasn't aware of.
   
    \item \textbf{Analyze the properties of possibly related attributes \textit{before} the join occurs:}
    Before joining a new attribute into a dataset, users will need to gather information about the new attribute to gauge the new attribute's impact on the quality of the dataset.
    For this reason, users should be provided with as much information as possible about the potentially joinable attributes before the join.
    This might include metadata, examples of the attribute, and any available information about the general distribution of the new data.  In addition, it is important to communicate whether there will be any missing values in the joined attribute.  
   

    \item \textbf{Specify aggregations:}
    The relationships between entities in the user's dataset and entities in the external data source are varied - they could be one-to-one, one-to-many, or many-to-many.
    In order to fit information encoded as plural relationships into a single row of data, aggregations must be specified.
    For example, a user might want to add the populations of a list of states into the their data. However, because there could be multiple measures of the state's population over the years, the user would need to perform an aggregation function over these results, such as \textsc{minimum}, \textsc{maximum}, or \textsc{average} depending on their analysis need.

    \item \textbf{Connect through to additional data:}
    Attributes relevant to the user's analysis goals may not always be directly encoded as a property of an entity that exists on the initial dataset.
    For example, a user may want to augment a dataset of countries with the population of each country's capital.
    This information may not be directly accessible as a property of each country in the data repository.
    Instead, population size is linked to the entity that represents the capital, which in turn is linked to the country.
    To support these cases, users should be supported to join \textit{through} intermediate attributes giving them fine-tuned control over how joins pass through multiple relationships.
   
   
   
   
   
\end{enumerate}

\DIFaddbegin \subsection{\DIFadd{Design Goals}}

\DIFadd{Based on this task analysis, we iterated through several sketches to come up with a set of design goals.  In the subsequent section, we describe a system with visualizations and interactions that address these goals.  However, this list of design goals should prove to have a broader impact than }Auger\xspace \DIFadd{alone - they can inform how data augmentation can be integrated into other visual analytics tools.
}

\begin{enumerate}[label=\textbf{G\arabic*:}, topsep=0pt, itemsep=0pt,parsep=3pt]
    \item \textbf{\DIFadd{Attributes as first-class citizen.}}  \DIFadd{Because external data is found based on attributes in the original data (}\textbf{\DIFadd{T1}}\DIFadd{), the list of attributes is the most important data to encode.  In early sketches, we used a data table to show the user what the data looked like, and allowed users to click on column headers to view new attributes and view joined data in enriched cells, similar to the }\textit{\DIFadd{ETable}} \DIFadd{from Kahng et al~\mbox
\cite{kahng_2016}}\hspace{0pt
.  We ultimately decided that a table encoding dedicated too much space to the table formatting and data found in the table cells.  Instead, we settled on a primary view that listed all attributes of the dataset.  By viewing the list of attributes, users are able to get a sense of what attributes might still be needed for their analysis.  This list can update as new attributes are foraged.  Additional details about each attribute, such as summary statistics and examples, can be provided on demand.
   
    }\item \textbf{\DIFadd{Visual exploration of joinable data:}} \DIFadd{In order to discover interesting joinable data (}\textbf{\DIFadd{T2}}\DIFadd{) and specify aggregations on that data to create new attributes (}\textbf{\DIFadd{T3}}\DIFadd{), users should be able to explore the set of joinable data.  Visual cues can help in this exploration, by encoding meaningful information about external data like the join quality or the data distribution.  As the dataset evolves, users should be given previews of the data they have added to enable iterations of exploration.
    }\item \textbf{\DIFadd{Provide visual examples for complex queries.}} \DIFadd{In practice, building complex queries that stepped through intermediate information on the knowledge graph becomes very complicated because it requires multiple aggregations over the joined data.  For these types of complex queries, visualizing the knowledge graph about one example in the data can help a user understand how they are combining and shaping the data.
}\end{enumerate}

\DIFaddend \section{Auger\xspace: A Visual Analytics System}

In this section, we describe the function and design of Auger\xspace.  \DIFdelbegin \DIFdel{We start by describing the }\DIFdelend \DIFaddbegin \DIFadd{The general workflow of }Auger\xspace \DIFadd{starts when a user uploads a tabular dataset.  The user can then search for related attributes of any column in their dataset containing entities that can be found in Wikidata.  They can augment their dataset with these related attributes by specifying aggregations.  After several iterations, if the user is satisfied with the augmented dataset, they can export their data for further analysis in other visual analytics tools.  
}

\DIFadd{In this section, we first describe the }\DIFaddend design of each component of the interface and their interactions.  Then, we \DIFdelbegin \DIFdel{give details on the implementation of the back-end connection to the Wikidata }\DIFdelend \DIFaddbegin \DIFadd{explain how these user interactions are translated into queries that can be executed over a }\DIFaddend knowledge graph.






\subsection{User Interface}

\DIFdelbegin \DIFdel{Here, we describe the general look and feel of 
\DIFdel{and go into depth about how its design enables the tasks listed above. As shown in }\DIFdelend
Figure~\ref{fig:system_overview}, Auger\xspace has the following main interface components:

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/through_join_small_with_labels.png}
    \vspace{-4mm}
    \caption{When a through join is specified, a dialog window pops up to allow the user to specify aggregations at each level of the join.  To help them specify that query to the system, Auger\xspace shows them an example of the neighborhood in the knowledge graph that is queried for a single row of the table.  In this image, the user would like to add the lowest life expectancy of any neighboring countries because they believe this will help their analysis of conflict.  And for each country, several values are available for life expectancy, so the user selects the ``max'' operator.  All six neighbors for Iraq, but only three are shown for simplicity of the illustration.  }
    \vspace{-3mm}

    \label{fig:through_join}
\end{figure}







 \vspace{3pt} \noindent \textbf{Column View:} This view shows the attributes and their data types (\DIFdelbegin \DIFdel{represented using colored icons) that are present in the loaded data (}\DIFdelend \DIFaddbegin \textbf{\DIFadd{G1}}\DIFadd{, }\DIFaddend see Figure~\ref{fig:system_overview}-A). \DIFdelbegin \DIFdel{As the user discovers new attributes, they can remove any particular column if they find it is not helpful to their analysis by a simple toggle button.  Users can click on any row of this view to see }\DIFdelend \DIFaddbegin \DIFadd{Users can view details on demand about each attribute, including }\DIFaddend the data distribution of that attribute shown as a histogram chart, \DIFdelbegin \DIFdel{along with any available meta-data describing that attribute (}\DIFdelend \DIFaddbegin \DIFadd{several example values, the type of unit (i.e. kilogram or mile, if applicable) (}\DIFaddend see Figure~\ref{fig:system_overview}-C).

 \vspace{3pt} \noindent \textbf{Related Attributes:} A user can \DIFdelbegin \DIFdel{toggle a search for a collection of }\DIFdelend \DIFaddbegin \DIFadd{search for }\DIFaddend related attributes of any \DIFdelbegin \DIFdel{string-like }\DIFdelend \DIFaddbegin \DIFadd{string }\DIFaddend attribute in the Column View\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T1}
\DIFdel{)}\DIFdelend .  Auger\xspace shows the \DIFdelbegin \DIFdel{retrieved attributes from the connected knowledge graph in a second column }\DIFdelend \DIFaddbegin \DIFadd{related attributes discovered from the knowledge graph }\DIFaddend next to the list of attributes of the current table, as seen in Figure~\ref{fig:system_overview}-B. 
\DIFdelbegin \DIFdel{Each row in this list represents a potential attribute that can be joined to the selected column.  Similar to attributes in the column view, metadata and distributions of each attribute can be viewed in a tooltip, as seen in Figure~\ref{fig:system_overview}-C.  The numbers surrounded by colored circles, also known as donut charts, represent the estimated percent of rows of the dataset that have joinable data for that attributein the knowledge graph.
}\DIFdelend
 \DIFaddbegin \DIFadd{An estimation of the join quality of each related attribute is encoded visually with a donut chart, and details on demand are also provided for each attribute, including distribution of values.
 }\DIFaddend This information can help the user identify whether a related attribute has the potential to help their analysis \textit{before} joining it to their data (\DIFdelbegin \DIFdel{Task }\DIFdelend \textbf{\DIFdelbegin \DIFdel{T2}\DIFdelend \DIFaddbegin \DIFadd{G2}\DIFaddend }).  Both the estimated probability and the estimated attribute distribution are calculated from random sample of possible joinable data, since calculating the true values would require completing the full join for every row in the dataset.  For more details on implementation techniques, see section~\ref{sec:technique}.

 \vspace{3pt} \noindent \textbf{Adding Attributes:} To add any attribute, users click the plus symbol, and Auger\xspace will show a drop-down menu revealing various \DIFdelbegin \DIFdel{join-operations users can perform to join this attribute to the data}\DIFdelend \DIFaddbegin \DIFadd{options for join operations}\DIFaddend . The available join operations correspond to the data type of the relationship.  If the relationship is one-to-one, i.e. a country has a single head of government, then a single value will be retrieved for each row of the dataset.  If the relationship is one-to-many or many-to-many, an aggregation must be specified.  For example, if the attribute is a collection of numbers, such as set of populations recorded for a country, the user can select numerical aggregations such as \DIFaddbegin \textit{\DIFadd{count}}\DIFadd{, }\DIFaddend \textit{mean}, \textit{max}, \textit{min}, \textit{sum}, or \textit{variance}\DIFdelbegin \DIFdel{(Task }\textbf{\DIFdel{T3}
\DIFdel{).  }\DIFdelend \DIFaddbegin \DIFadd{.  If the attribute is a datetime, the user can select from }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{max}}\DIFadd{, or }\textit{\DIFadd{min}}\DIFadd{.  If the attribute is a string, the user can select either }\textit{\DIFadd{count}} \DIFadd{or }\textit{\DIFadd{through}}\DIFadd{, a type of placeholder operation that let's the user step through this attribute to form complex multi-hop joins (more details are provided below).  In addition to these join operations, for any type of data, the user can select to randomly sample from the collection.  This operation is useful if an attribute is generally expected to have only one value per entity, but due to data quality issues, some rows might have it recorded twice, like date of birth.  
 }\DIFaddend 

 \DIFdelbegin \DIFdel{It can be very costly to resolve all of the queries needed to join an additional attribute to an entire dataset ; gathering the }\DIFdelend \DIFaddbegin \DIFadd{Extracting information for the whole dataset from the knowledge graph is too computationally demanding to be performed at interactive rates.  Gathering the }\DIFaddend data from the knowledge graph requires the retrieval of many different parts of the knowledge graph \DIFdelbegin \DIFdel{rather than }\DIFdelend \DIFaddbegin \DIFadd{in contrast to }\DIFaddend the retrieval of a single column from a relational database.  Auger\xspace mitigates much of that time cost by only joining attributes for the top 10 rows of the dataset.  Users are still able to glance at the dataset preview, as seen in Figure~\ref{fig:system_overview}-D, and get an idea of the data that is being joined to the dataset \DIFaddbegin \DIFadd{(}\textbf{\DIFadd{G2}}\DIFadd{)}\DIFaddend .  This lets the user explore the available attributes rapidly.  When the exploration phase is completed, the full join can be executed.  

 \vspace{3pt} \noindent \textbf{Through Joins:} In some cases, the user may want to join to data through an intermediate attribute.  If the intermediate attribute has a one-to-one relationship with the rows of the original dataset, i.e. a country has one head of government, then the user can simply join the intermediate attribute first, and then use that as their starting point to seek more related attributes.  If the intermediate attribute is a one-to-many relationship, then the user must specify multiple levels of aggregation.  We call this type of aggregation a \textit{``multi-hop aggregation,''} since it requires aggregating data across multiple hops on the knowledge graph.  

 As an example, suppose the user wants to determine if a country has a lower life expectancy than its neighbors.  Consider how that would be calculated for a single country, Iraq.  To gather the required data from the knowledge graph, the user must first join to all bordering countries, then connect through them to reach the life expectancies of those countries.  Once all data is selected, aggregations must be specified to produce the desired value.  In Wikidata, countries have multiple life expectancies recorded, so the user has to specify that they want the maximum life expectancy recorded.  That value is then calculated for each bordering country (i.e. Iran, Turkey, Jordan, etc.).  Then, they have to specify that they want the minimum among all bordering countries.  Expressing this type of query is complicated enough to explain for a single value, let alone for an entire column.  
 \DIFaddbegin 

 \DIFaddend In Auger\xspace, the user is shown a simplified illustration of the topology of the knowledge graph to assist them in understanding the aggregations that the user must choose, as seen in Figure~\ref{fig:through_join}.  \DIFaddbegin \DIFadd{For a given row of the dataset, up to three values of the intermediate attribute and the target attribute are sampled from the knowledge graph.  The system then displays these values on a graph, representing a small slice of the knowledge graph.  Users can then dynamically set the aggregations (i.e. }\textit{\DIFadd{count}}\DIFadd{, }\textit{\DIFadd{mean}}\DIFadd{) that occur at each level of the join graph, and see the resulting value that would get joined for that row.  }\DIFaddend By viewing the values that will get \textit{passed through} the knowledge graph to ultimately calculate the attribute for one row of their data, users get reinforcement that the complex query they are building \DIFdelbegin \DIFdel{results in reasonable data.
 }\DIFdelend \DIFaddbegin \DIFadd{in }Auger\xspace \DIFadd{matches the query they are building in their head (}\textbf{\DIFadd{G3}}\DIFadd{).
 }\DIFaddend 

\DIFdelbegin
\textbf{\DIFdel{Showing the augmented data: }}
\DIFdel{When additional attributes are successfully added to the data, the column view shows a card on the right showing the list of attributes that are added to the data. Added columns are also shown as a nested list in the first card below the parent attribute through which it was retrieved from the knowledge graph, as seen in Figure~\ref{fig:system_overview}-A. If the added column is a string, it can then be used as a starting point to search for more related attributes.  In that manner, the user can reach further and further away from the initial attribute set by iteratively adding attributes As attributes are added, the table view updates with the new list of columns and shows the user example values for the top 10 rows of the dataset (see Figure \ref{fig:system_overview}-D).

}\DIFdelend




 









\subsection{Backend Implementation}

\label{sec:technique}

Here, we describe how our visual analytics system augments a dataset by translating the tasks supported by the user interface described above into valid queries over a knowledge graph.  Auger\xspace gathers data from the connected knowledge graph in two different subroutines.  In all examples in the paper, Auger\xspace connects to Wikidata using the SPARQL query language.\footnote{Example queries generated by the system for both subroutines are available in the supplemental material.}



\vspace{3pt} \noindent \textbf{Finding Related Attributes:} This subroutine receives a column of data as input \DIFdelbegin \DIFdel{(i.   e. the Country columnand all it's data, }\DIFdelend \DIFaddbegin \DIFadd{and returns a list of attributes along with assorted metadata.   Consider a dataset with a Country column, i.e. }\DIFaddend ``Germany,'' ``France,'' ``Austria,'' etc.\DIFdelbegin \DIFdel{), and returns a list of attributes along with assorted metadata.  
In }\DIFdelend \DIFaddbegin \DIFadd{.
Unlike a column in a SQL table, where entities and their relations might be expected to adhere to a schema, a list of countries in }\DIFaddend a knowledge graph \DIFdelbegin \DIFdel{, although these are all entities about a country, they }\DIFdelend might not share the same set of attributes\DIFdelbegin \DIFdel{(i. e. they might have different numbers of edges connected to the entities, and those edges might describe different types of relationships).  
}\DIFdelend \DIFaddbegin \DIFadd{. For example, in Wikidata, some Western countries, such as Germany, can have more than a thousand attributes, while other countries have only a fraction of that.  
}\DIFaddend In order to provide a user with a consistent set of attributes for these entities, we first need to find their commonalities.
As such, the primary goal of this subroutine is to return the list of attributes that are available on as many of the entities as possible.  
\DIFaddbegin 

\DIFaddend Checking the related attributes of each entity to gather this list can take prohibitively long, especially if the input data contains thousands to millions of rows.  \DIFdelbegin

\DIFdelend As an optimization step to make this subroutine support a user's interactive analysis, we employ a sampling-based technique to reduce computation time. With this optimization, some subset of the dataset is randomly sampled; \DIFdelbegin \DIFdel{empirically }\DIFdelend \DIFaddbegin \DIFadd{in our experiments }\DIFaddend we have found that 20-30 rows are sufficient \DIFaddbegin \DIFadd{to differentiate high-quality attributes from poor-quality ones}\DIFaddend . Each sampled row is mapped to an entity on the knowledge graph, and a list of related attributes for that row is retrieved.  Then, the lists for all the rows are compared, and the 50 attributes that appear on the most lists are returned.  \DIFdelbegin \DIFdel{The percent of lists that the attribute appears on is passed on as metadata, and is visually encoded in 
\DIFdel{as a colored donut chart, seen in each attribute row in Figure~\ref{fig:system_overview}-B.
    

\DIFdel{Various metadata is returned, including three examples for each attribute.  To get the data for the histograms, seen in Fig~\ref{fig:system_overview}-C, an additional query is run for each of }\DIFdelend \DIFaddbegin \DIFadd{In this query, we also gather all details of }\DIFaddend the \DIFdelbegin \DIFdel{50 related attributes to get 20-30 examples through which to get a rough estimate of the distribution of the data.This subroutine runs queries in parallel and is only limited in speed by the concurrency limits on the knowledge graph API. }\DIFdelend \DIFaddbegin \DIFadd{attributes that are displayed on demand, such as the data distribution of that attribute shown as a histogram chart, several example values, and the type of unit (i.e. kilogram or mile, if applicable).  }\DIFaddend For the types of exploration done in the user study and in examples given in this paper, retrieving the list of related attributes from Wikidata typically takes 2-5 seconds.

\DIFdelbegin
\DIFdelendFL
    \DIFaddbeginFL 



\begin{figure}[ht]
    \DIFaddendFL \centering
    \DIFdelbeginFL
\DIFdelendFL \DIFaddbeginFL \includegraphics[width=0.45\textwidth]{figures/acled_gdp_fatalities.png}
        \DIFaddendFL \vspace{-3mm}

    \caption{\DIFdelbeginFL \DIFdelFL{Three steps of analysis of an armed conflicts dataset.  Visualizations show }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Visualizing }\DIFaddendFL the number of records \DIFdelbeginFL \DIFdelFL{of each event type }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{in the ACLED dataset in April 2018 }\DIFaddendFL by \DIFdelbeginFL \DIFdelFL{(a) country}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{per capita GDP}\DIFaddendFL , \DIFdelbeginFL \DIFdelFL{(b) basic form of government, and (c) government type}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{a field extracted from Wikidata}\DIFaddendFL .  \DIFdelbeginFL \DIFdelFL{Iniitally, only country is available.  
\DIFdelFL{is used to gather basic form }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Magnitude }\DIFaddendFL of \DIFdelbeginFL \DIFdelFL{government first}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{marks encodes number of fatalities}\DIFaddendFL , \DIFdelbeginFL \DIFdelFL{then, after some analysis by the user, it is used to gather the government type to better understand patterns in the }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{while color encodes }\DIFaddendFL event \DIFdelbeginFL \DIFdelFL{types}\DIFdelendFL \DIFaddbeginFL \DIFaddFL{type}\DIFaddendFL .    \DIFdelbeginFL \DIFdelFL{Visualizations }\DIFdelendFL \DIFaddbeginFL \DIFaddFL{Visualization }\DIFaddendFL generated with Tableau. }
        \vspace{-3mm}
    \label{fig:tableau_use_case}
\DIFdelbeginFL
\DIFdelend \DIFaddbegin \end{figure}
\DIFaddend 

    
\vspace{3pt} \noindent \textbf{Materializing Joins:} This subroutine takes in join instructions and returns an augmented dataset with an additional column.  The join instructions specify the path taken through the knowledge graph, along with any aggregation functions, such as ``count'' or ``max.''  Auger\xspace builds a SPARQL query to crawl the knowledge graph to retrieve the desired data (see Fig~\ref{fig:teaser}).  The time to retrieve data scales linearly with the number of rows and is limited by the concurrency limits on the knowledge graph API. While the user is interactively exploring attributes, Auger\xspace will only materialize the join for the top 10 rows of the data table to provide the user with a preview of the table in real time (see Fig~\ref{fig:system_overview}-D).  When the foraging is done, all joins are materialized for the entire dataset (rather than just a sample of rows) in the order in which they were constructed by the user, which takes around 10-15 seconds on the datasets explored in the user study in section~\ref{sec:user_study}.




Auger\xspace is implemented with a VueJS frontend web application and a NodeJS backend to handle all asynchronous calls.  Queries are sent concurrently whenever possible.  Source code is \DIFdelbegin \DIFdel{withheld to maintain anonymity and will be released in the final publication}\DIFdelend \DIFaddbegin \DIFadd{available at }\url{https://github.com/TuftsVALT/snowcat}\DIFaddend .





\DIFaddbegin \vspace{-0.5mm}
\DIFaddend 

\label{sec:system}


\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 1: Conflict Data}



To demonstrate the value of column augmentation  for insight generation, we present a \DIFdelbegin \DIFdel{use case }\DIFdelend \DIFaddbegin \DIFadd{usage scenario }\DIFaddend in which Auger\xspace is integrated into a typical workflow for the popular visual analytics tool, Tableau.   Suppose a political scientist wants to study the factors that lead to armed conflict.  They download the Armed Conflict Location \& Event Data Project (ACLED) dataset for April 2018, which contains \DIFdelbegin \DIFdel{hundreds of }\DIFdelend \DIFaddbegin \DIFadd{2,279 }\DIFaddend records of armed conflicts across many different countries and \DIFdelbegin \DIFdel{times}\DIFdelend \DIFaddbegin \DIFadd{dates}\DIFaddend \cite{raleigh2010introducing}.  Initially, they load the data into Tableau to analyze relationships between countries\DIFdelbegin \DIFdel{and event types.  They produce a heatmap, seen in Figure~\ref{fig:tableau_use_case}(a), to try to detect patterns, but }\DIFdelend \DIFaddbegin \DIFadd{, event types, and fatalities stemming from armed conflict.  They see no clear pattern that emerges as they }\DIFaddend find it difficult to visually group countries without more data.

\DIFaddbegin 

\DIFaddend They load their dataset into Auger\xspace to look for additional attributes that may be relevant to event types.  When the system is first loaded, the user sees a list of the attributes in the uploaded table in Figure~\ref{fig:system_overview}-A.  They also can see the first five rows of the data below in Figure~\ref{fig:system_overview}-D.  

The political scientist believes \DIFdelbegin \DIFdel{the type of government could be related to the event type}\DIFdelend \DIFaddbegin \DIFadd{there might be economic metrics that effect the types and severity of conflicts}\DIFaddend , so they click the \textit{related attributes} button next to ``Country'' in Figure~\ref{fig:system_overview}-A.  The system returns a list of related attributes found by scraping Wikidata, as seen in Figure~\ref{fig:system_overview}-B (\textbf{T1}).  The user scans through the list of related attributes, looking for information about the \DIFdelbegin \DIFdel{type of government}\DIFdelend \DIFaddbegin \DIFadd{countries' economies}\DIFaddend .  They see that Wikidata holds the \textit{\DIFdelbegin \DIFdel{basic form of government}\DIFdelend \DIFaddbegin \DIFadd{nominal GDP per capita}\DIFaddend } for each country.  
\DIFdelbegin \DIFdel{The tooltip for that attribute, seen in Figure~\ref{fig:system_overview}-C, provides more information to affirm that this holds the right data by showing the user examples of the attribute, like ``unitary state,'' ``federal republic,'' and ``absolute monarchy'' (}\textbf{\DIFdel{T2}
\DIFdel{).  The user }\DIFdelend
\DIFaddbegin \DIFadd{The user }\DIFaddend adds this attribute to the dataset.  While looking for \DIFdelbegin \DIFdel{the form of government}\DIFdelend \DIFaddbegin \DIFadd{economic data}\DIFaddend , the user also notices additional interesting \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend for analysis and joins them for future analysis, including the max \textit{inflation rate}, minimum \textit{life expectancy}, \DIFdelbegin \DIFdel{mean }\DIFdelend \DIFaddbegin \DIFadd{the }\DIFaddend \textit{\DIFdelbegin \DIFdel{nominal GDP per capita}\DIFdelend \DIFaddbegin \DIFadd{type of government}\DIFaddend }, and the mean \textit{Human Development Index} (\textbf{T3}).

The user loads this next iteration of the dataset into Tableau, and generates \DIFdelbegin \DIFdel{the same visualization but with the basic form of government replacing the country name, }\DIFdelend \DIFaddbegin \DIFadd{various visualizations including the event types and fatality numbers with their newly-discovered attributes.  As can be }\DIFaddend seen in Figure~\ref{fig:tableau_use_case}\DIFdelbegin \DIFdel{(b).  While some patterns begin to emerge, the user notes that the basic form of government is too fine-grained for the type of analysis they want to do, since most categories only have one or two countries in them.  They return to 
\DIFdel{to see if there is any higher-level categorization available for that column by searching for attributes off of that attribute (}\textbf{\DIFdel{T4}
\DIFdel{).  They find that Wikidata holds a }\textit{\DIFdel{subclass of}}
\DIFdel{attribute for basic forms of government with examples including ``democracy, '' ``monarchy,'' ``republic,'' and add that attribute using a through join before returning to Tableau to generate the final visualization, seen in Figure~\ref{fig:tableau_use_case}(c).  Now, they can see a pattern in that conflicts concentrate on republics in the region, with democracies and monarchies having interesting patterns in their types of conflicts as well.  This use case }\DIFdelend \DIFaddbegin \DIFadd{, grouping event types by nominal GDP per capita reveals a trend that certain event types, like }\textit{\DIFadd{Remote violence}} \DIFadd{and }\textit{\DIFadd{Battle - No change of territory}}\DIFadd{, appear to have more severe fatalities in countries that have lower nominal GDP per capita.  In contrast, other event types like }\textit{\DIFadd{Battle - Government regains territory}} \DIFadd{and }\textit{\DIFadd{Battle - Non state actor overtakes territory}} \DIFadd{don't seem to be very different between high nominal GDP per capita countries and low ones.  The additional visual analysis enabled by the added attribute, nominal GDP per capita, has led the user to new hypotheses that can then be further explored.  This usage scenario }\DIFaddend demonstrates how the integration of column augmentation into a traditional insight generation pipeline can unlock new analyses in-situ.

\section{\DIFdelbegin \DIFdel{Use Case }\DIFdelend \DIFaddbegin \DIFadd{Usage Scenario }\DIFaddend 2: Modeling \DIFdelbegin \DIFdel{Unemployment}\DIFdelend \DIFaddbegin \DIFadd{Poverty}\DIFaddend }

\label{sec:use_cases_ml}
\DIFdelbegin

\DIFdelFL{A modified version of 
\DIFdelFL{used for both Use Case 2 and Task 2 of the user study. Users can build predictive models as they augment their dataset to help them discover attributes that help predict a target variable.  This figure portrays the construction of regression models to predict the rate of unemployment in counties of the United States.  The history of trained models is shown at the bottom of the column view.   By mousing over a model, the user can see the five highest feature importance scores for that model, and the list of attributes used to train that model are highlighted in black.}}
   
\DIFdelend

In our second \DIFdelbegin \DIFdel{use case}\DIFdelend \DIFaddbegin \DIFadd{usage scenario}\DIFaddend , we demonstrate the value of \DIFdelbegin \DIFdel{Auger }\DIFdelend \DIFaddbegin Auger\xspace \DIFaddend for a separate type of visual analytics task - the generation of a predictive model.  Many visual analytics systems have been constructed to enable domain experts to interact with and steer the generation of machine learning models on their data.  Augmenting a dataset with new \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend is one way in which a user can imbue domain knowledge into the modeling process.  They may know that certain attributes can help the prediction while others may mislead.  While it may seem unusual to augment a training set with additional attributes, the additional attributes found on the knowledge graph can likewise be added to any data that the resulting model is asked to predict on at test time when the model is deployed.

\DIFdelbegin \DIFdel{We }\DIFdelend \DIFaddbegin \DIFadd{To demonstrate the use of CAVA in the model building process, we }\DIFaddend modify Auger\xspace to be able to train a predictive model at any point in the data augmentation process\DIFdelbegin \DIFdel{(see Figure~\ref{fig:ml_use_case}). 
When the user instructs 
\DIFdelend \DIFaddbegin \DIFadd{. 
Users can instruct the system }\DIFaddend to build a \DIFdelbegin \DIFdel{model, }\DIFdelend \DIFaddbegin \DIFadd{predictive model with the current version of the data.  }\DIFaddend Auger\xspace splits the data into a training and test set in the ratio of 0.8:0.2 and trains a random forest regressor.  For each model, Auger\xspace reports the \textit{R Squared} score on the training and test sets, as well as the feature importance scores of the five most important features.  \DIFaddbegin \DIFadd{This functionality was designed to be representative the iterative workflow of model builders, and to examine the ways }Auger\xspace \DIFadd{could help in-situ model building.
}\DIFaddend

Consider Andy\xspace is a public policy analyst who seeks to build a regression model on \DIFdelbegin \DIFdel{the unemployment rate dataset \mbox
\cite{unemployment2015} }\hspace{0pt
with $1200$ data samples from the year $2015$. Every row in the data is a county and contains a sparse set of attributes such as }\textit{\DIFdel{County}
\DIFdel{, }\DIFdelend \DIFaddbegin \DIFadd{a U.S. Census Bureau dataset containing data about poverty in different counties around the U.S.\mbox
\cite{poverty2020}}\hspace{0pt
.
The dataset contains $3136$ rows, and $6$ attributes--the dataset index, }\textit{\DIFadd{FIPS}}\DIFadd{, }\DIFaddend \textit{State}, \textit{\DIFdelbegin \DIFdel{Month}\DIFdelend \DIFaddbegin \DIFadd{County}\DIFaddend }, \DIFdelbegin \DIFdel{and the response variable }\textit{\DIFdel{Unemployment Rate}
\DIFdelend \DIFaddbegin \textit{\DIFadd{RUCCode}}\DIFadd{, and the number of residents living in the county under the poverty line}\DIFaddend .
Andy\xspace seeks to augment this data by adding meaningful columns to accurately predict the \textit{\DIFdelbegin \DIFdel{Unemployment Rate}\DIFdelend \DIFaddbegin \DIFadd{Poverty}\DIFaddend }. 
When Andy\xspace first loads the data in Auger\xspace, an initial model is trained with an \textit{R Squared} score of \DIFdelbegin \DIFdel{$0.646$ }\DIFdelend \DIFaddbegin \DIFadd{$0.577$ }\DIFaddend on training and \DIFdelbegin \DIFdel{$0.292$ }\DIFdelend \DIFaddbegin \DIFadd{$-1.801$ }\DIFaddend on test set respectively\DIFaddbegin \DIFadd{; indicating that the model badly overfitted the training data}\DIFaddend . 
Andy\xspace seeks to improve the regression model's performance further by augmenting the base data using Auger\xspace's workflow and visual interface.
Andy\xspace searches for related attributes of the variable \textit{State} from the Column View. In response, Auger\xspace shows a list of related attributes that Andy\xspace may consider to add to the data. 
From these attributes they click on a set of interesting attributes such as \textit{shares border with}, \textit{\DIFdelbegin \DIFdel{head of government}\DIFdelend \DIFaddbegin \DIFadd{life expectancy}\DIFaddend }, \textit{inflation rate}, etc. to see the distribution of values, and its metadata as a text view. 

From these set of attributes Andy\xspace thinks that the attributes \textit{Inflation Rate} and \textit{\DIFaddbegin \DIFadd{nominal }\DIFaddend GDP per capita} are good attributes to predict \DIFdelbegin \DIFdel{the }\DIFdelend \textit{\DIFdelbegin \DIFdel{unemployment rate}\DIFdelend \DIFaddbegin \DIFadd{Poverty}\DIFaddend } and thus decides to add them to the data using the join operation \textit{\DIFdelbegin \DIFdel{Median}\DIFdelend \DIFaddbegin \DIFadd{Mean}\DIFaddend }. They notice that the newly added column is displayed on the Table View. However looking at a few rows of the table Andy\xspace finds quite a few cells that are empty indicating that the joined data may have missing values. 
Nonetheless, Andy\xspace clicks a button to construct a new regression model using the augmented data.
Andy\xspace notices that the \textit{R Squared} score marginally improves (new score: \DIFdelbegin \DIFdel{$0.712$ and $0.301$ }\DIFdelend \DIFaddbegin \DIFadd{$0.586$ and $-1.810$ }\DIFaddend on training and test set respectively). They hover the mouse over the model metric card to see the list of ``Top $5$'' attributes with their weights utilised in the regression model. 
While Andy\xspace expected to see a substantial improvement in the model performance, 
they infer that the marginal improvement is probably due to missing values in the data after the join operation. \looseness=-1

Motivated to improve the model further, Andy\xspace searches for columns that may directly help to predict \DIFdelbegin \DIFdel{the unemployment rate }\DIFdelend \DIFaddbegin \DIFadd{poverty }\DIFaddend per county. Based on prior knowledge, Andy\xspace understands that the \DIFdelbegin \DIFdel{population }\DIFdelend \DIFaddbegin \textit{\DIFadd{population}} \DIFaddend of a state may be directly proportional to \DIFdelbegin \DIFdel{the unemployment rate}\DIFdelend \DIFaddbegin \DIFadd{its poverty}\DIFaddend , and they add that attribute. 
In the process of searching for other relevant columns, Andy\xspace notices the column \textit{Maximum temperature}. Inquisitive to see if a temperature of a state is correlated with its \textit{\DIFdelbegin \DIFdel{unemployment rate}\DIFdelend \DIFaddbegin \DIFadd{Poverty}\DIFaddend } they add it to the table.
After constructing a new regression model, Andy\xspace notices that the \textit{R Squared} score \DIFdelbegin \DIFdel{improved substantially from $0.712$ to $0.855$ and from $0.301$ to $0.338$ }\DIFdelend \DIFaddbegin \DIFadd{changed from $0.586$ to $0.577$ and from $-1.810$ to $0.541$ }\DIFaddend for the training and test dataset respectively. Happy with the progress so far they decide to remove any column that may not be contributing to the prediction task. First\DIFaddbegin \DIFadd{, }\DIFaddend they remove the \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{columns }\DIFaddend \textit{\DIFdelbegin \DIFdel{Month}\DIFdelend \DIFaddbegin \DIFadd{RUC Code}\DIFaddend }\DIFaddbegin \DIFadd{, and }\textit{\DIFadd{nominal GDP per capita}} \DIFaddend (by triggering the slider on the Column View) and then triggers Auger\xspace to construct a new regression model. 
As expected Andy\xspace notices that the \textit{R Squared} score did not change. 
\DIFdelbegin \DIFdel{Next to investigate the contribution }\DIFdelend
\DIFaddbegin \DIFadd{Next }Andy\xspace \DIFadd{searches for related attributes }\DIFaddend of the column \textit{\DIFdelbegin \DIFdel{Maximum temperature
\DIFdel{, they remove it from the data.

\DIFdel{notices that the }\textit{\DIFdel{R Squared}}
\DIFdel{score of the new model dropped marginally meaning that the column }\textit{\DIFdel{Maximum temperature}}
\DIFdel{was helpful in improving the model.
They }\DIFdelend \DIFaddbegin \DIFadd{County}}\DIFadd{.
They explore the set of choices shown in the Column View.
They choose to }\DIFaddend add  the column \DIFdelbegin \DIFdel{back to the data}\DIFdelend \DIFaddbegin \textit{\DIFadd{County-population}} \DIFadd{(by median operation), and }\textit{\DIFadd{shares border with}} \DIFadd{(by count operation). }Andy\xspace \DIFadd{constructs a new regression model to see the models' train and test }\textit{\DIFadd{R Squared}} \DIFadd{score improved considerably ($0.807$, and $0.746$ respectively). Content with the improvement, they export the model and the data to continue analysis}\DIFaddend .

In a short time, Andy\xspace has constructed a regression model with substantially better R2 score than the model trained on the base data.  They have also gained insight into which attributes are relevant to their modeling problem, which may help their understanding of the resulting machine learning model.

\DIFdelbegin


\DIFdelend \section{Preliminary User Study}
\label{sec:user_study}
We evaluate Auger\xspace in a preliminary user study. The purpose of the evaluation is to validate Auger\xspace both in terms of its usability and its effectiveness in helping a user improve a machine learning model through data augmentation. 
Specifically, we hypothesize that:

\begin{itemize}
    \item \textbf{H1}: Auger\xspace allows users to accurately join external data given a written description of that data. 
   
   
    \item \textbf{H2}: Auger\xspace is able to help users discover additional data that can improve the predictive quality of machine learning models trained on the dataset.
   
\end{itemize}

We recruited $6$ participants (3 Female, 3 Male), between the age of $23-36$. We required each participant to have at least an elementary knowledge of machine learning and data analysis. Due to COVID-19, we were not able to conduct an in-person study. Instead  we conducted an online study using Bluejeans\footnote{https://www.bluejeans.com/}. The participants interacted with Auger\xspace on their own computer while sharing their screen.
We provided the participants with a url to our system that was hosted on our local machine that is exposed using the Ngrok remote tunneling software\footnote{https://ngrok.com/}. The study took approximately $50-60$ minutes and we compensated the participants with a $\$10$ Amazon gift card.


\subsection{Study Design}
Before the study we asked participants to fill out a background information questionnaire regarding their name, age, gender, machine learning expertise and various use cases in which they use machine learning.
We began the study by showing the participants a tutorial video of Auger\xspace, explaining the workflow, interface GUI elements, and its interaction capabilities that support various join operations. Next we asked the participants to perform three tasks, the first of which was a practice task to ensure that they were 
sufficiently knowledgeable about Auger\xspace to perform the experimental trials. We proceeded to the experimental sessions only when we observed that the participants were confident and able to use Auger\xspace on their own. In the next two tasks we asked the participants: (1) To add a set of specified \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend to a given dataset. These \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend can be added to the data using various join operations supported by Auger\xspace. (2) To freely augment the data such that they can improve the performance metric of a machine learning model (metric being a R Squared Score of a regression model). 
We used the Scikit Learn machine learning library \cite{scikit_ml} to construct \textit{Random Forest Regression} models, in the same manner as described in Section~\ref{sec:use_cases_ml}.
We collected the following data from each experimental trial: (1) \textit{Task competition time}, (2) \textit{Task Accuracy}, (3) \textit{Model performance metric} e.g., R Squared Score, and (4) \textit{User ratings} collected using a post-study Likert-scale questionnaire.


\subsection{Datasets}

For the tutorial video and practice task we used \DIFdelbegin \DIFdel{a }\DIFdelend \DIFaddbegin \DIFadd{the same }\DIFaddend U.S. Census Bureau dataset \DIFdelbegin \DIFdel{containing data about poverty in different counties around the U.S.\mbox
\cite{poverty2020}}\hspace{0pt
.
The dataset contains $3136$ rows, and $6$ attributes such as }\textit{\DIFdel{FIPS}
\DIFdel{, }\textit{\DIFdel{State}
\DIFdel{, }\textit{\DIFdel{County}
\DIFdel{, }\textit{\DIFdel{RUCCode}
\DIFdel{, and the number of residents living in the county under the poverty line.
}\DIFdelend \DIFaddbegin \DIFadd{as section~\ref{sec:use_cases_ml}.
}\DIFaddend For the first task in our experimental session we use the IMDB Movies dataset\cite{imdb2020} containing $500$ movies and $28$ attributes such as 
\textit{Director-name}, \textit{Duration}, \textit{Movie-Title}, \textit{Movie Cast Facebook Likes}, etc. The second task in the experimental session which required users to augment data and construct regression models, used the unemployment rate dataset\cite{unemployment2015}. This dataset contained $1200$ rows where each row corresponded to a county's poverty rate measured in a given month. This dataset contained only $5$ \DIFdelbegin \DIFdel{attributes such as }\DIFdelend \DIFaddbegin \DIFadd{attributes--the dataset index, }\DIFaddend \textit{Month}, \textit{State}, \textit{County}, and the \textit{Unemployment-Rate} (dependent variable). 

\subsection{Tasks and procedure}
Participants were first asked to complete a practice task.  
During the practice task, their answers were not recorded and their performance was not included in the analysis described below.  
For the practice task, using the Poverty dataset, we asked the participants to add three \DIFdelbegin \DIFdel{columns}\DIFdelend \DIFaddbegin \DIFadd{attributes}\DIFaddend : (1) the area of the county, (2) the population of the state, and (3) the earliest inception date of any county that each county shared borders with. 
While the first two \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend could be retrieved using a straight-forward \textit{join by value} operation, the third \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend required the \textit{join through} operation to search for the required \DIFdelbegin \DIFdel{column }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend that was one ``hop'' away in the knowledge graph. For the first experimental task using the IMDB dataset, participants were given descriptions of four attributes to add: (1) the director's country of citizenship, (2) the number of awards received by the director, (3) the number of cast members in each movie, (4) the sum of the number of awards received by all the producers of each movie.  
Note that attributes 3 and 4 required the participants to \textit{join through} several attributes.  

The final experimental task gave participants $10$ minutes 
to augment the Unemployment rate dataset with as many attributes as they'd like that they believed would aid in building an accurate regression model to predict poverty rate.  
At any point in the 10 minutes, participants could instruct Auger\xspace to build a regression model with the current dataset to give them insight into whether they were improving the predictive modeling process.  
Building a model took about 10-20 seconds, and participants would be shown the change in R Squared Score for each new model, as well as the  weights of top ``5'' \DIFdelbegin \DIFdel{features }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend used in the model. 
Random Forest regression models were used because their flexibility, speed, and accuracy met the constraints of the experiment.


\subsection{Data Collection}

After all the tasks were completed, participants were asked to fill out a post-study questionnaire (using Google Forms) that included: (1) Likert-scale\DIFaddbegin \DIFadd{\mbox
\cite{likert1932technique} }\hspace{0pt
}\DIFaddend rating questions on their use of the system, (2) a NASA-TLX\cite{hart1988development} questionnaire to measure task difficulty, and (3) Open-ended descriptive questions asking users about the interface design, the workflow, and other responses related to improve the usability of the system\footnote{Pre-experiment and Post-experiment surveys are attached as supplemental material}. 
The Likert-scale questions asked the participants to rate (in a scale of $1-7$, $1$ being ``strongly disagree'') if they found the system: (1) Easy to learn, (2) Intuitive, and (3) Expressive for data augmentation. 
We used the open-ended descriptive feedback to qualitatively evaluate the usability and the interaction design of Auger\xspace for their tasks.
Open-ended prompts included (1) \textit{Describe your thoughts about Auger\xspace}, (2) \textit{Describe things you disliked about the system or the workflow. Elaborate on how you think it could have been improved}, and (3) \textit{Describe your strategy or your process to augment the data for Tasks 1 and 2}\DIFaddbegin \DIFadd{.
}\DIFaddend With the consent of the participants,
each session was \DIFaddbegin \DIFadd{both }\DIFaddend video and audio-recorded. We encouraged the participants to verbalize their thoughts following a think-aloud protocol.
Furthermore, to assess if data augmentation using Auger\xspace led to any change in the regression models performance\DIFaddbegin \DIFadd{, }\DIFaddend we saved: (1) model metric (i.e., R Squared Score), (2) \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend weights, and (3) predicted values from the model. We also saved user mouse-clicks to analyze the set of \DIFdelbegin \DIFdel{columns }\DIFdelend \DIFaddbegin \DIFadd{attributes }\DIFaddend the user explored to augment the data.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/questionnaire_responses_2.pdf}
    \vspace{-8mm}
    \caption{Participants' responses about the system (Qn1--Qn4, the higher the better) and the overall effort (Qn5--Qn9, the lower the better).}
        \vspace{-3mm}

    \label{fig:survey_ratings}
\end{figure}


\subsection{Result and Analysis}
\DIFdelbegin \DIFdel{We report three types of results: task performance, user satisfaction, and qualitative feedback.
}\DIFdelend

\subsubsection{Task Performance}

To assess \textbf{H1}, we reviewed the number of attributes that participants correctly joined in Task 1.
Five out of six participants properly joined all four expected attributes.
Only one participants missed one attribute when joining the number of awards received by the director.
This indicates that in general participants were able to use Auger\xspace to accurately add new data attributes.

Next, to assess \textbf{H2}, we similarly reviewed the \textit{R Squared} score improvement that participants achieved by augmenting additional data in Task 2.
On average, participants improved the \textit{R Squared} Score by $0.048$ ($\sigma=0.014$), starting with a baseline \textit{R Squared} Score of $0.292$. 
The area and population of the county are two most frequently joined attributes across all the participants, which is reasonable as these two attributes are usually the most influential for predicting unemployment rate. 

As a result of our analysis, we \textbf{accept \DIFdelbegin \DIFdel{both }\DIFdelend H1\DIFdelbegin \DIFdel{and H2}\DIFdelend } because all participants were able to successfully use Auger\xspace to explore and identify the correct attributes for data joining in Task \DIFdelbegin \DIFdel{1, and }\DIFdelend \DIFaddbegin \DIFadd{1.
However, we }\textbf{\DIFadd{neither accept nor reject H2}} \DIFadd{as the study results did not contain clear evidence of users being able }\DIFaddend to improve the predictive quality of a regression model in Task \DIFdelbegin \DIFdel{1. 
}\DIFdelend \DIFaddbegin \DIFadd{2 beyond trivial improvements.
}\DIFaddend 



\subsubsection{User satisfaction}

We use the post-study questionnaire to assess user satisfaction of the data augmentation process in Auger\xspace.
\autoref{fig:survey_ratings} shows the results of the questionnaire responses.
Although the participant size is too small to infer statistical significance using quantitative analysis, we do observe that participants found Auger\xspace easy and intuitive to use based on the likert scale user ratings (\autoref{fig:survey_ratings}, \textbf{Qn1,2}).
Participants were also satisfied with the two main functionalities provided by Auger\xspace, finding relevant attributes from the data and joining additional data (\autoref{fig:survey_ratings}, \textbf{Qn3,4}).
The mean ratings of \textbf{Qn1--4} were all 6 or above.
Furthermore, in analyzing user satisfaction related to the overall process (\autoref{fig:survey_ratings}, \textbf{Qn5--9}), we found that participants were generally satisfied with the process of conducting the tasks, as the mean ratings of \textbf{Qn5--9} are all 2 or lower.


\subsubsection{Qualitative Feedback}

To assess \textbf{H1} and \textbf{H2} from a qualitative perspective, we analysed participants’ descriptive feedback collected from post-study interviews. We also observed participants' workflow in using Auger\xspace from audio and video recordings of their computer.
We report the following three main qualitative user responses from the study:

\vspace{3pt} \noindent \textbf{Intuitive workflow:}
All the participants \DIFdelbegin \DIFdel{fount }\DIFdelend \DIFaddbegin \DIFadd{found }\DIFaddend Auger\xspace's workflow intuitive to find and augment relative data attributes.
The primary justification for Auger\xspace's intuitiveness is that Auger\xspace provides a visual interface that is easy to infer and for users to search and add relevant attributes from the data.
As \textit{P6} noted, ``\textit{I liked the interface of this tool because it was easy to navigate and add/remove columns to the base data}.''
The intuitiveness helped most participants to easily perform the requested tasks, especially Task 1.
For example, \textit{P1} described his experience in doing Task 1 as ``\textit{just looked at the question and implemented it}'' and ``\textit{it was pretty straight forward}.''

\vspace{3pt} \noindent \textbf{Two major strategies in model construction:}
When conducting Task 2, we observed two common strategies used by the participants: (1) searching for relevant attributes based on existing or prior knowledge and (2) exploring all possible combinations of available attributes to find one that results in improvement in model performance.
Some participants relied on their existing domain knowledge for augmenting the data with new attributes.
This was summarized nicely by \textit{P5}, ``\textit{I was actually using my domain knowledge. Thinking which factors can improve unemployment rate predictions, and then tried to select the variables based on the choices I had. This actually helped me improve the model's performance}.''
However, a few participants who may lack prior knowledge about the data, instead explored the dataset and tried out all possible combinations of attributes to find a model that is a significant improvement over the model trained on the base data.
``\textit{I selected columns one by one to see the change of R-squared score. By doing so, I was able to filter the variables that decrease R-square. In the end, I was able to get a relatively high R-squared model}'' (\textit{P6}).
The rest of the participants adopted a combination of both strategies.
For example, \textit{P2} described his strategies as follows, ``\textit{Thought a bit about what could boost performance, then I did some trial and error (manual feed-forward selection of attributes)}.''


\vspace{3pt} \noindent \textbf{Need for more powerful system features for expert users:}
Participants with visual analytics backgrounds expected editable visualizations to represent or encode \DIFdelbegin \DIFdel{feature }\DIFdelend \DIFaddbegin \DIFadd{attribute }\DIFaddend distribution differently.
``\textit{It would be cool to add additional ways of visualizing attributes}'' (\textit{P2}).
What's more, participants with database backgrounds wanted more details about queries of fetching data to increase flexibility.
``\textit{Maybe consider providing the real query of fetching the dataset to the expert users to give them more clear sense and allow them to change the query}” (\textit{P1}).
\DIFdelbegin \DIFdel{Similarly, participants with more expertise in machine learning sought more control over the model building process.
``}\textit{\DIFdel{May be add hyperparameter tuning for the model building phase, such as L1, L2 norm or change learning algorithm of the model}
\DIFdel{'' (}\textit{\DIFdel{P2}
\DIFdel{).
}\DIFdelend
\DIFdelbegin \DIFdel{All these }\DIFdelend \DIFaddbegin \DIFadd{These }\DIFaddend suggestions are valuable for guiding our further improvement in generalizing Auger\xspace to tackle various problem domains.



\subsection{Limitations}

While our user study results \DIFdelbegin \DIFdel{support both }\DIFdelend \textbf{\DIFaddbegin \DIFadd{support }\DIFaddend H1} \DIFdelbegin \DIFdel{and }\DIFdelend \DIFaddbegin \DIFadd{but }\DIFaddend \textbf{\DIFaddbegin \DIFadd{neither accept nor reject }\DIFaddend H2}, the results should be read in the light of the study limitations.
The number of the participants for our study are limited, which might be a confounding point of the results.
As we conducted the study online, the uncertainty existing in the online setup, such as internet connection, might also confound the results.
To improve the performance of the system and the fluidity of user interaction, we limited the number of related attributes that participants could fetch for each parent attribute.
Although it did help participants with a smoother user experience during the study, participants also complained about the restriction as it limited their performance of tasks, especially Task 2.
Despite the limitations, the user study does help us to better identify the limitations in our system design and improve the system with new features.





\section{Discussion}
\label{sec:discussion}

Through our user study and \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we have demonstrated the efficacy of information foraging using knowledge graphs within visual analytics systems.  The generality of Auger\xspace suggests that data augmentation could \DIFaddbegin \DIFadd{be }\DIFaddend added into traditional visual analytics system workflows to improve the outcome of any embedded task.  Many fruitful avenues of research arise when considering how to apply information foraging to the full space of \DIFdelbegin \DIFdel{use cases }\DIFdelend \DIFaddbegin \DIFadd{usage scenarios }\DIFaddend which are served by visual analytics systems.

\subsection{Mental Models of Data Augmentation}

Knowledge graphs can be difficult to reason about, especially when they are used to find data corresponding to an entire column of a dataset.  In Auger\xspace, we address this by showing the user previews of the joined dataset, as well as visualizations of the distribution of joined data and an example of the portion of the knowledge graph that is used to construct a single value (see Figure~\ref{fig:through_join}).  This approach was based on our conversations with designers of visual analytics applications for building predictive models.  A better understanding of the user's mental model of the knowledge graph and the joining process is needed to generalize this approach to other \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend .  We generally aimed to hide the complexity of the underlying knowledge graph, but it may be the case that more direct exploration of the knowledge graph is helpful for some cases.

\DIFaddbegin \subsection{\DIFadd{Sparsity in Knowledge Graphs}}

\label{sec:sparsity}

\DIFadd{In this work, we rely on the high quality of Wikidata to find attributes that have sufficient support for the user's dataset.  But in many cases, only 10-15\% of the rows of a dataset will have joinable data for a given attribute.  Because knowledge graphs do not hold schematic data, they can suffer from data sparsity issues.  We feel that the experiments in this work show that there are many popular data types, including geographic entities like countries or states, that have sufficient robust data to discover attributes with full support on a user's dataset.  In situations where the attribute of interest is only present for a small percent of rows of the dataset, we suggest two possible solutions.  First, the additional data can still be useful as additional information injected into the dataset, and using imputation or setting reasonable default values (using a tool such as the }\textit{\DIFadd{data tamer}} \DIFadd{~\mbox
\cite{Stonebraker2013} }\hspace{0pt
can make that information usable by downstream analyses.  Second, a user can explore the reduced dataset in which the sparse attribute is present, and if it results in promising analysis, they can use }Auger\xspace \DIFadd{to search for other more complete attributes that might be correlated with that sparse attribute, based on their domain expertise.  Lastly, we would like to point out that knowledge graphs are not read-only structures, and }Auger\xspace \DIFadd{may help point out where it would be valuable for an organization to invest in recording more data.
}

\DIFaddend \subsection{Design Space for Interactions with Knowledge Graphs}

In Auger\xspace, the channel between the user and the knowledge graph is limited to the two subroutines shown in section~\ref{sec:technique}, which allow the user to see lists of related attributes and then construct queries given that information.  However, there is more potential to improve the user's control over the process and address edge cases by expanding the set of interactions between user and knowledge graph.  Automated processes in Auger\xspace could be replaced by collaborations between user and system.  

For example, ambiguities in the entity resolution used to retrieve a list of related attributes can be solved by user interaction.  A list of countries could refer to the governmental entities they describe, or they could refer to the national soccer teams participating in the World Cup; there will always exist cases where disambiguating the type of \DIFdelbegin \DIFdel{a column }\DIFdelend \DIFaddbegin \DIFadd{an attribute }\DIFaddend will require domain expertise from a user.  

The user could also benefit from more fine-grained control over the process of building aggregation queries.  Users may want to join timestamped data, which would necessitate some specification from the user of how to parse and interpret temporal columns.  Spatial data offers an additional potential, as the user might want to use geographical data in their dataset to search for the closest weather station or other geographically tagged entity.  There are many types of queries that might necessitate different user interactions than have been used in previous visual analytics systems.


\DIFdelbegin \subsection{\DIFdel{Scalability}}
\addtocounter{subsection}{-1
\DIFdelend

\DIFdelbegin \DIFdel{In order for 
\DIFdel{to operate at the speed necessary for exploration, it makes use of only samples of the data, rather than the entire dataset .  When gathering the list of related attributes for a column, it samples only a subset of values from that column.  Those related attributes are then shown with estimations of their distribution as well as what percent of rows have a joinable value.  While we use a sampling-based approach, there is plenty of opportunity to develop methods that provide better estimates or allow for joining to more data in less time.  
The database community is working on tools for query optimization over knowledge graphs\mbox
\cite{Mohamed2020}}\hspace{0pt
, which will help.  But specific indices, query optimizations, and data structures should be explored by the visual analytics community, the same way that many advancements in querying tabular data have been made to allow for faster data exploration\mbox
\cite{lins2013nanocubes, wang2018neuralcubes, pahins2016hashedcubes, liu2013immens, tao2019kyrix}}\hspace{0pt
.
}\DIFdelend






\section{Conclusion}
\label{sec:conclusion}
In this work, we presented Auger\xspace, a visual analytics system for explorative information foraging using knowledge graphs.  Knowledge graphs offer a wealth of information on a broad array of topics that could be used to improve the outcome of embedded tasks of visual analytics systems.  In our \DIFdelbegin \DIFdel{use cases}\DIFdelend \DIFaddbegin \DIFadd{usage scenarios}\DIFaddend , we demonstrated that the data gathered through Auger\xspace could result in better predictive models and better insight generation on two datasets.  And in our experiment, we showed that Auger\xspace is simple to learn and use, as all six of our participants were able to effectively explore and join data on multiple datasets within a 60 minute session.  


\DIFdelbegin

\DIFdelend \DIFaddbegin \acknowledgments{
We thank our collaborators in DARPA's Data Driven Discovery of Models (d3m) program, as well as the reviewers for their helpful feedback.  This work was supported by National Science Foundation grants IIS-1452977, OAC-1940175, OAC-1939945, DGE-1855886, and 1841349, as well as DARPA grant FA8750-17-2-0107.
}
\DIFaddend
\bibliographystyle{abbrv-doi}

