A European manufacturing giant would like to pilot orchestrating data pipelines using Managed Airflow running on Azure. [https://www.astronomer.io/]. A large migration program will follow a successful Pilot.

Job Description

Building and orchestrating data pipelines using Managed Airflow [https://www.astronomer.io/] on Azure. It’s a pilot project to demonstrate the benefits of using Apache Airflow as an open-source platform for developing, scheduling, and monitoring batch-oriented workflows.

Responsibilities

• Design and implement a limited number of workflows, orchestrating data pipeline, using Apache Airflow & dbt
• Design consistent log overview and validate the capability of Airflow to handle large data volumes
• Measure solution performance and ease of use
• Recommend transition [migration] scenarios

Skills And Qualifications

• Demonstratable experience with Apache Airflow [using Astro is a major plus]
• Experience with dbt
• Experience with Azure
• Experience with SQL and Git.
• Experience in developing, deploying, and supporting scalable data pipelines (including but not limited to ETL, streaming ingestion, and processing)
• Knowledge of Databricks is a major plus
• Well-versed with Agile methodology and DataOps