Google Cloud Data Catalog — Live Sync Your On-Prem Hive Server Metadata Changes
Disclaimer: All opinions expressed are my own, and represent no one but myself…. They come from the experience of participating in the development of fully operational sample connectors, available at: github.

The Challenge

Entering the big data world is no easy task, the amount of data can quickly get out of hand. Look at Uber story, on how they deal with 100 petabytes of data using the Hadoop ecosystem, imagine if every time they would sync their on-premise metadata into a Data Catalog, a full run was executed, that would be impractical.

We need a way to monitor changes executed at the Hive server, and whenever a Table or Database is modified we capture just that change and incrementally persist it in our Data Catalog.

If you missed the last post, we showcased ingesting on-premise Hive metadata into Data Catalog, in that case we didn’t use an incremental solution.

To grasp the situation, a full run with ~ 1000 tables took almost 20 minutes, even if only 1 table had changed. In the Uber story, that would be no fun, right?
['Marcelo Costa']
['Database', 'Big Data', 'Google Cloud Platform', 'Apache Hive', 'Data']