Associate Data Engineer / Data Engineer

Responsibilities

• Design and implement scalable, reliable distributed data processing frameworks and analytical infrastructure. 

• Be part of a team to define, design, and implement data integration, management, storage, consumption, backup, and recovery solutions that ensure the high performance of the organization's enterprise data. 

• Develop Structured Query Language (SQL), Data Definition Language (DDL), and Python or equivalent programming scripts to support data pipeline development, problem-solving, data validation, and performance tuning. 

• Work with software engineers, devops, ML engineers, and data scientists to achieve the organization’s goals. 

Requirements

• Minimum 2-5 years experience in the similar capacity with BS/MS degree in Computer Science, Engineering or a related subject

• Strong experience with data engineering tools and platforms, including Delta Lake and Data Lakehouse architectures

• Proficiency in data pipeline orchestration tools, such as Apache Airflow.

• Broad knowledge of programming languages is required (Python/Scala).

• Understanding about data warehousing solutions, relational database theories and no-sql databases.

• Good knowledge of new and emerging tools for extracting, ingesting and processing of large datasets (Kafka, Spark, Hadoop, DataBricks or equivalent).

• Familiarity with Azure Data Factory and Azure Logic Apps is a big plus.

• Hands-on experience with Amazon Web Services is a big plus and knowledge about other cloud systems (Azure/GCP).

• Understanding about dockerization and kubernetes, data modeling and design patterns is a big plus.

• Knowledge of web scraping technologies is a big plus (selenium, beautiful soup etc).

• Familiarity with Linux.

• Excellent interpersonal, communication and organizational skills are required. 

Please email your CV with two non-related referees to: jobs@ilabs.lk www.incubatelabs.com|Tel: 0112177251 | 240, 32 Denzil Kobbekaduwa Mw, Battaramulla