Data Engineer

Responsibilities

Managing and maintaining data pipeline

• End-to-end pipeline orchestration, data schedule and automate pipeline runs, ensuring data is processed at the right intervals, whether it’s batch processing or real-time streaming.
• Set up mechanisms to collect data pipelines that outline the flow of data from source to destination
• Set up mechanisms to collect data from variety of sources, which can include databases, external APIs, log files, IoT devices, and more.
• Understand data extraction processes such as ETL, ELT, CT, and CDC to retrieve data.
• Understand dbt (data build tool) to manage data quality, data documents, and dependencies.
• Strongly understand Airflow and dbt (data build tool) to manage data quality, data documents, and dependencies.
• Monitoring and error handling to ensuring that data flows smoothly and efficiently.
• Strongly understand data modern stack trends.
• Understand how to use Confluent or Kafka for data streaming pipeline.
• Understand CI/CD pipelines.

Data Optimization

• Performance optimization for SQL Script.
• Strong programming language skills in SQL and Python are required
• Web scrapping analysis.
• DBT (data build tool) for data transformation and data modelling.
• Logging and Monitoring.

Data Integrity and Compliance

• Ensuring the quality, accuracy and consistency of data by implementing data validation checks and monitoring data sources for anomalies.
• Managing data retention and archiving policies to determine how long data should be retained and when it should be archived.
• Managing data governance including compliance to local country regulations e.g. PDPA and data security.

Design Documents

• Clarify and understand the functionality of each data job.
• Document data processes, configurations, and best practices and communicate effectively with team members and stakeholders.
• Data lineage and dbt to represent dataflow.

Requirements:

• Bachelor's degree in Computer Science, Computer Engineering, Quant/Maths
• Experience in SAP is a plus.
• Experience in Machine Learning is a plus.
• Experience with the Cloud services especially on Google Cloud services.
• Good communication both Thai & English.
• Understand MLOps is a plus
• Highly motivated and proactive with ability to work under pressure to meet demanding deadlines and able to work independently.
• “Can do” spirit.
• Team Player.
• Eager to learn new services.