Intermediate Airflow Developer

iVedha Inc. — Canada, CA
Published on

About the Role:

We are looking for an experienced Intermediate Airflow Developer with over 2 years of experience to help transition our existing Windows scheduler jobs to Apache Airflow DAGs. In this role, you’ll play a critical part in modernizing and optimizing our task automation processes by converting existing jobs into efficient, manageable, and scalable workflows in Airflow. You will also work on security hardening, implement data pipelines, and design ETL processes.

Key Responsibilities:

  • Convert Windows Scheduler Jobs to Airflow: Migrate existing Windows-based scheduled jobs into Airflow DAGs, ensuring smooth execution and reliability.
  • Develop and Optimize DAGs: Author, schedule, and monitor Directed Acyclic Graphs (DAGs) for data workflows, ETL tasks, and automation processes.
  • Programming and Scripting: Utilize Python as the primary language for creating Airflow DAGs and task logic, along with SQL for data manipulation.
  • Set Up and Configure Airflow: Provide instructions and configurations for setting up Airflow environments, including resource allocation and high availability.
  • Security Hardening: Implement security best practices, including role-based access control and data protection.
  • Data Pipelines and ETL: Design data pipelines to transform and aggregate data while ensuring its accuracy.
  • Troubleshooting and Optimization: Monitor DAGs for performance issues and optimize for reliability.
  • Documentation and Knowledge Transfer: Document migration steps and configurations for future reference.

Requirements:

  • 2 years of experience with scheduling tools and task automation.
  • Proficiency in Apache Airflow with experience in authoring and managing DAGs.
  • Strong Python skills for creating data workflows and ETL processes.
  • Proficiency in SQL for data manipulation within ETL tasks.
  • Familiarity with Windows Scheduler.
  • Security awareness in the context of Airflow.
  • Strong analytical skills for troubleshooting and optimizing workflows.
  • Effective communication skills for documenting processes.

Preferred Qualifications:

  • Experience with Airflow in cloud environments like AWS, GCP, or Azure.
  • Exposure to Docker and containerization.
  • Knowledge of DevOps practices and CI/CD tools.
  • Familiarity with additional programming languages such as Bash, PowerShell, or JavaScript.
← All jobsApply