Partners HealthCare


Companies like Partners HealthCare
are looking for tech talent like you.

Sign up to start matching for free.

About Partners HealthCare

Job Description





* We are looking for a self-motivated Data Engineer to join our data engineering team.
* Design, Develop, construct, test and maintain architectures such as Data Lake, large-scale data processing systems
* Big data ecosystem related Tool selection and POC analysis
* Gather and process raw data at scale that meet functional / non-functional business requirements (including writing scripts, REST API calls, SQL Queries, etc.)
* Develop data set processes for data modeling, mining and production
* Integrate new data management technologies (Collibra, Informatica DQ..) and software engineering tools into existing structures
* The candidate will be responsible for participating in building new Data Lake, expanding and optimizing our data platform and data pipeline architecture, as well as optimizing data flow and collection for cross functional teams.
* The ideal candidate is an experienced data pipeline builder who enjoys optimizing data systems and building them from the ground up.
* The Data Engineer will support our Software Developers, Database Architects, Data Analysts and Data Scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects.
* The right candidate will be excited by the prospect of optimizing and/or re-designing our data architecture to support next generation of products and data initiatives.Principal Duties and Responsibilities
* Create and maintain optimal data pipeline architecture, assemble large, complex data sets that meet functional / non-functional business requirements on Hadoop and relational data systems
* Build the Hadoop infrastructure required for optimal extraction, transformation, and loading of data from traditional/legacy data sources.
* Work with stakeholders including the Management team, Product owners, and Architecture teams to assist with data-related technical issues and support their data infrastructure needs.
* 3-5 Years of experience building and optimizing 'big data' data pipelines, architectures and data sets.
* Advanced hands-on SQL knowledge and experience working with relational databases for data querying and retrieval.
* Experience with big data frameworks/tools: Hadoop, Kafka, Spark, etc.
* Experience with relational SQL and NoSQL databases, including MS SQL Server, Hive, HBase.
* Experience with data security.
* Experience with building processes supporting data transformation, data structures, metadata, dependency and workload management.

Experience with Java and/or Python a plus.

* Hadoop based technologies (e.g. hdfs, Spark). Spark Experience is must
* Strong SQL skills on multiple platform (preferred MPP systems)
* Database Architectures
* Data Modeling tools (e.g. Erwin, Visio)
* 5 years of Programming experience in Python, and/or Java
* Experience with Continuous integration and deployment
* Strong Unix/Linux skills
* Experience in petabyte scale data environments and integration of data from multiple diverse sources
* Kafka, Cloud computing - Azure, AWS; machine learning, text analysis, NLP & Web development experience is a plus
* Healthcare experience, most notably in Clinical data, Epic, Payer data and reference data is a plus but not mandatory

Skills Required

* Expertise in the Hadoop Data Lake and relational Data Warehouse platforms
* Demonstrated experience in Hadoop big data technologies (Cloudera, Hortonworks), Data Lake development
* Experience with real time data processing and analytics products
* Experience with at least one Cloud based data technologies (AWS, Azure, GCP)
* Cloudera or Hortonworks certification preferred
* Cloud certification would be preferred
* Large data warehousing environments in at least two database platforms (Oracle, SQL Server, DB2, etc)
* Programming experience in Python, Java, SQL, good to have .Net, C#
* ETL, data processing expertise in Hadoop (map-reduce, spark, sqoop) and SSIS, HealthCatalyst, Informatica
* Familiarity with data governance and data quality principles, good to have experience with data quality tools
* Ability to independently troubleshoot and performance tune in large scale data lake, enterprise systems
* Knowledge of data architecture principles, data warehousing, agile development, DevOps methodologies
* Understanding of change management techniques, and the ability to apply them
* Excellent verbal and written communication skills, problem solving and negotiation skills
* Act as an effective, collaborative team member

About Partners HealthCare


6947 employees
Partners HealthCare

800 Boylston St 11th Floor

Let your dream job find you.
