We will not be providing hackathon pre-packaged utilities like containers or synthetic data, please bring your own computers, data, software, knowledge!
The days will include intensive sessions related to OHDSI CDM and OHDSI tools, hands on experts to help debug or solve your Hadoop issues, Atlas on Hadoop, hands on building ETL code, as well as Hadoop 101 training if beginners attend.
A beginner’s course on Hadoop presented by experts at Cloudera. This session is designed for attendees who may be skilled at OHDSI but beginners to the 20+ tools in the Apache big data ecosystem, or interested in understanding the four or five Apache open source software projects being used by OHDSI members in lieu of Oracle, SQL Server, Postgres or Redshift.
Part A – Using the Synpuf data, demonstrate how ETL code can be written in Hadoop. This class will go through how to set up the environment needed and review the ETL code written by Odysseus for the OMOP conversion.
Part B – Breaking out into groups, refactor the ETL code written to improve performance and other issues identified. Outcome of this group is to publish a version of the ETL code that can be used by all member to convert Synpuf data on Hadoop in their environment.
As a group, talk through what are some of the lessons learned during OMOP conversions using Hadoop. This will cover from the project initiation, architecture set up, ETL mapping, ETL coding, to automation. How can we as a group standardize and share work? As an outcome of this working group, publish at least 10 standard guidelines that the community can use when developing ETL OMOP conversions in Hadoop. These guidelines will not be ‘official’ but will be submitted to the OHDSI community for debate. If time permits, work toward a conversion ‘specification’ or ‘cookbook’ or other tools instantiating those guidelines that might help OHDSI members who need to do a conversion could be created.
Bring your Hadoop/Spark/SOLR OHDSI issues or questions to the F2F. Curious about what tools might e available to help perform conversion/analytics in Hadoop? Whether on OMOP CDM conversion, hardware, cloud, other topics, we will have staff from Cloudera and other organizations that will help you with your Hadoop and other issues. Partnered with each Cloudera staff will be other members of the community who has been doing OMOP CDM conversions for a while just in case you have a question related to OMOP CDM, as Cloudera professionals onsite will have no knowledge of CDM model or SQLRender.
Sign-ups can be for either 30 minutes or 1 hour sessions.
Discuss the issues the community faces when using the ODHSI tools on Hadoop. A perfect example is Atlas. What are the issues preventing ATLAS/WebAPI generated analytical code from working with Impala-enabled OMOP CDM. Take the issues and beginning fixing the issues one by one. If issues cannot be fixed, create a backlog and roadmap on how to fix the remaining issues.
As a group, talk through what are some of the lessons learned during OMOP conversions using Hadoop. This will span from project initiation, architecture set up, ETL mapping, ETL coding, to automation and export. How can we as a group standardize and share work? As an outcome of this working group, publish at least 10 standard guidelines that the community can use when developing ETL OMOP conversions in Hadoop. These guidelines will not be ‘official’ but will be submitted to the OHDSI community for debate. If time permits, work toward a conversion ‘specification’ or ‘cookbook’ or other tools instantiating those guidelines that might help OHDSI members who need to do a conversion could be created.
Bring your Hadoop/Spark/SOLR OHDSI issues or questions to the F2F. Whether on OMOP CDM conversion, hardware, cloud, other topics, we will have staff from Cloudera and other organizations that will help you with your Hadoop and other issues. Partnered with each Cloudera staff will be other members of the community who has been doing OMOP CDM conversions for a while just in case you have a question related to OMOP CDM, as Cloudera professionals onsite will have no knowledge of CDM model or SQLRender.