We are seeking a highly motivated Data Engineer to join our growing team. In this role, you will play a critical part in designing, developing, and maintaining our data infrastructure on the Azure Cloud platform. You will leverage your expertise in PySpark and Python to build efficient and scalable data pipelines that ingest, transform, and load data from various sources.
Responsibilities:
- Design, develop, and implement data pipelines using Azure data services (ADLS, Data Factory, Synapse Analytics, etc.) and PySpark.
- Collaborate with data scientists and analysts to understand data requirements and design data models.
- Write and maintain high-quality, efficient, and maintainable Python and PySpark code.
- Ensure data quality through data cleansing, transformation, and validation processes.
- Monitor and optimize data pipelines for performance and scalability.
