Data driven organizations require automated management and governance across all of their data without being restricted to a single data format, data platform or even a single vendor’s offerings. ODPi Egeria (https://odpi.github.io/egeria/) is a new open source project that provides the open APIs, event formats, types and integration logic to make this possible.
The first release of Egeria focuses on creating a single virtual view of metadata. It can federate queries across different metadata repositories and has the ability to synchronize metadata between different repositories. The synchronization protocol controls what is shared, with which repositories and ensures that updates to metadata can be made with integrity.
Before delving into the details of Egeria, it is worth level-setting what we mean by metadata.
Metadata describes data in all its forms. This includes where the data is located, how it is stored, how frequently it is changing, what it represents, how it is organized, who owns it, how accurate it is, who is using it and how it should be governed. It is the knowledge base that sits underneath all data activities.
A key deliverable of Egeria is its comprehensive definition of open metadata types (https://odpi.github.io/egeria/open-metadata-publication/website/open-metadata-types/) that provides the structures used to exchange metadata.
Metadata management is the process of maintaining metadata. Often data tools and platforms come with their own metadata repository and management services. These metadata repositories are for the exclusive use of their accompanying tools and reflects the data known about by their users. The effect is that there are typically many metadata repositories operating in an organization, each with a different view of the data that the organization has access to.
The role of Egeria is to allow data tools and platforms to continue to work with their own metadata repositories, whilst ensuring the metadata used across the organization is consistent.
The metadata repositories are registered into what is called an open metadata repository cohort. An open metadata repository cohort has common event topic on an event bus such as Apache Kafka (http://kafka.apache.org/). Each repository that is to share metadata puts a registration event on this topic. This registration event provides information about the repository including its network address. The other repositories pick up the network address to configure their local federation service. This federation service is used to query metadata from each of the repositories to create an enterprise view. The event topic is also used to replicate selected metadata between the repositories.
The OMRS (https://odpi.github.io/egeria/open-metadata-implementation/repository-services/) is the component within ODPi Egeria that supports the federation and metadata exchange services. It is integrated with each repository (https://odpi.github.io/egeria/open-metadata-publication/website/open-metadata-integration-patterns/) and manages all of the open metadata protocols.
And this is only the start of Egeria’s journey. Stay tuned as we add open governance capability for Chief Data Officers, Privacy Officers, Data Owners, Data Scientists and much more. Join us to get involved at the Egeria Github page.
ODPi will be hosting an Open Metadata & Governance Workshop at Open Source Summit 2018, a co-located event for attendees to learn more about Egeria—a new project from the ODPi that supports the free flow of metadata between different technologies and vendor offerings. Egeria enables organizations to locate, manage and use their data more effectively. Egeria is built on open standards and delivered open source (Apache 2.0 licensed).
Join the Egeria Workshop in Vancouver on Tuesday, August 28 from 9:30 am – 5:30 pm, as industry leaders from ODPi will provide an overview of Egeria. See demonstrations of Egeria and Atlas in action; take a deep dive into the code; and share your own ideas and needs during collaboration and brainstorming in a preview of the Egeria roadmap.
The workshop will conclude with a networking event, where attendees can meet the experts and socialize over pizza and drinks. Attendance is free to conference registrants (sign up & add to your OSS schedule here). Feel free to drop-in for single session of interest… or take advantage of the full day workshop.
Click here for more information, including a detailed agenda.
Learn more about the Open Source Summit here.
The success that ODPi has achieved as a nonprofit organization committed to simplification and standardization of the big data ecosystem is driven by the dedication of our member organizations and individuals. Today we are introducing the first interview in a series of conversations with key ODPi contributors, exploring why they participate in ODPi. We seek to learn more about the individuals whose efforts are accelerating the development of today’s Big Data ecosystem, standards and solutions.
We recently sat down with Scott Gnau, chief technology officer of Hortonworks, to discuss his involvement with ODPi and what value Hortonworks gains from being an active member. Scott has spent his entire career in the data industry, and as Chief Technology Officer for Hortonworks he is currently responsible for the overall technology vision for Hortonworks, overseeing the company’s Engineering, Product Management and Support organizations.
ODPi: Thanks for joining us today Scott. Can you start by sharing a little background on your involvement with ODPI?
Scott: I’ve been involved with ODPi since the founding many years ago, when at the time I worked for a different company that was one of the original founding members of ODPI. Ultimately, the whole notion behind ODPi is to drive a broader sense of community and a level of standardization around some of the core technology in our space. Ultimately that’s a really good thing. If we can make a common kernel for software developers, it makes it easier for them to build their applications and take advantage of the technology that’s being developed as part of the larger Hadoop community. ODPI helps to do it with a level of confidence & compatibility.
ODPi: So has your role been with ODPI? Tell us about how you and Hortonworks have gotten involved with contributions, collaboration and achievements of the organization?
Scott: As for our contributions at Hortonworks, we invest in ODPi not only through the dedicated people who participate, but also certainly with the common code contributions that are donated and then managed by the foundation. Our participation with ODPi is really an extension of how we feel it’s important to invest in the community and to get involved in collaborative involvement in the community.
We’re well known for being an open source company. Our membership with ODPi is not really just about being open source, but also about how we can foster a sense of community that is open, with contributions to the effort coming from all over the industry. That collaboration is what I think helps make our market function properly, and ODPi is one way that represents how Hortonworks invests in that community.
ODPi: What is the real value of the work that ODPi is driving, and what’s the importance of providing a vendor-neutral home for these efforts?
Scott: The big thing is that with any project of this nature, you look at what’s going on in the market and try to respond and/or stay ahead. One of the cool things that we’re working on inside of ODPI now is a much larger focus on the whole notion of data governance, and support of Apache Atlas. When I think about how ODPI has helped the industry, ODPi drove consolidation of all the different distributions of Hadoop and created some standards in the market. That’s a really good thing, from a compatibility perspective as well as support and all those things.
That work largely being completed now, we think the next big thing the industry needs to tackle is data governance. Data governance is still a largely unsolved problem, and that’s with known, structured, transactional kinds of data. The world we live in now—with connected sensors, IoT and big data—brings highly-unknown and highly-variable data that may have even been created outside of the corporate firewall. So the notion of data governance becomes not only more important as we need to know what is the data, where did it come from, who has access to it… but it also becomes a more complicated problem to solve.
We really think that it’s important for Hortonworks to leverage our position in the community and work with the community at large to help create some semblance of standards-based approaches for metadata tagging and governance across this wide variety of data. ODPi is a really great place to do that because it includes multiple different vendors, some of whom compete with each other, coming together to help define standard constructs that can be leveraged across the industry.
ODPi: What excites you personally regarding the technical work being done in ODPi this year?
Scott: The notion of data governance is a big deal. There is a lot of really cool technology emerging, just look at the interesting algorithms being built in deep learning, or the whiz-bang use cases. In the end none of those solutions are sustainable if data is not trusted, and we can’t prove out that lineage. So I’m very excited about the current ODPi focus, governance is a hot button for me. While it may not seem like the sexiest thing out there, it’s the core component that’s going to make all of those interesting use cases viable over the long term.
I couldn’t be more enthusiastic about the value ODPi has created since its inception, and the support we’re getting from ODPi and its member community. From creating a semblance of common kernel components inside the core Hadoop stack, to the much more robust current focus on data governance, ODPI as a non-profit, neutral party is positioned to continue having a great impact.
Sign up for our Newsletter to receive the latest ODPi news and updates.