Title: Minimally-Supervised Attribute Fusion for Data Lakes,
Abstract: Aggregate analysis, such as comparing country-wise sales versus global market
share across product categories, is often complicated by the unavailability of
common join attributes, e.g., category, across diverse datasets from different
geographies or retail chains, even after disparate data is technically ingested
into a common data lake. Sometimes this is a missing data issue, while in other
cases it may be inherent, e.g., the records in different geographical databases
may actually describe different product 'SKUs', or follow different norms for
categorization. Record linkage techniques can be used to automatically map
products in different data sources to a common set of global attributes,
thereby enabling federated aggregation joins to be performed. Traditional
record-linkage techniques are typically unsupervised, relying textual
similarity features across attributes to estimate matches. In this paper, we
present an ensemble model combining minimal supervision using Bayesian network
models together with unsupervised textual matching for automating such
'attribute fusion'. We present results of our approach on a large volume of
real-life data from a market-research scenario and compare with a standard
record matching algorithm. Finally we illustrate how attribute fusion using
machine learning could be included as a data-lake management feature,
especially as our approach also provides confidence values for matches,
enabling human intervention, if required.