Data Engineer
IT Chapter — Canada, CA
- Published on
About the Role
The Data Engineer will be responsible for leading the delivery of data on a data and analytics platform. These services are at the forefront of building out data engineering practice using cloud-native technologies. The Senior Data Engineer must have experience in leading, designing, implementing, and collaborating with stakeholders to achieve the best results for our clients.
Responsibilities
- Proven design, build, and implementation of batch and real-time data pipelines.
- Driven by automated repeatable delivery of data that aligns to enterprise data governance standards.
- Experience in developing and proposing data models that conform to requirements.
- Ensuring the proposed design optimally addresses access and query patterns, data consumption, and adheres to internal architecture standards.
- Collaborate with various stakeholders across the business, including data scientists and IT.
- Build relationships and refine data requirements to meet various data and analytics initiatives.
- Increase data onboarding speed to the Data and Analytics platform.
- Build robust data pipelines for larger data consumption.
- Enhance the overall quality of data pipeline development through DevSecOps.
About the Candidate
Skills Required:
- Programming experience in Spark using modern languages such as Scala.
- Experience with modern data architectures like Azure Data Lake Storage, Azure Databricks, Azure Synapse, and Delta Lakes.
- Proven experience leading Data Engineering principles within an organization/team.
- Expertise in database modeling techniques: Data Vaults, Star, Snowflake, 3NF, etc.
- Experience working with streaming data architecture and technologies for real-time data: Spark Streaming, Kafka, Flink, Storm.
- Familiarity with relational and non-relational database technologies: SQL Server, Oracle, Cassandra, MongoDB, CosmosDB, HBase.
- Experience with source code management environments such as Azure DevOps, Git, Maven, and Nexus.
Assets:
- Experience within an Azure environment with strong Scala and Spark expertise.
- Demonstrated ability in modernizing data platforms and managing projects that develop Data Vaults.