Data Engineer

Responsibilities

• Lead the design and implementation of data solutions using Databricks SQL Databricks Workflows and PySpark.
• Oversee the development of scalable and efficient data pipelines to support business intelligence and analytics.
• Provide technical expertise and guidance to the development team to ensure best practices are followed.
• Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
• Ensure data quality and integrity by implementing robust data validation and cleansing processes.
• Optimize data processing workflows for performance and scalability.
• Develop and maintain documentation for data solutions including architecture diagrams and technical specifications.
• Conduct code reviews and provide constructive feedback to team members to improve code quality.
• Stay up-to-date with the latest industry trends and technologies to ensure our data solutions remain cutting-edge.
• Participate in the evaluation and selection of new tools and technologies to enhance our data capabilities.
• Provide support and troubleshooting for data-related issues to ensure timely resolution.
• Work closely with data scientists and analysts to enable advanced analytics and machine learning initiatives.
• Contribute to the continuous improvement of our data architecture and processes.

Qualifications

• Experience in Cloud ETL development activities using Informatica Cloud IICS including data quality data reconciliation
• Strong SQL experience
• Experience in Databricks Snowflake Python is preferred
• Cloud computing using Azure technologies - Azure SQL Azure SQL DWH Synapse ADLS ADF Cosmos DB is preferred