APPLYDJINN
Vacancy catalog
EPAM
Open role

Lead Python Data Engineer

EPAMKazakhstan
Work model
Office
Experience
5+ years
Employment
Not specified
Compensation
Not disclosed
Technology signal
12 tags

Technology context

12

Parsed from the vacancy text; ordered by relevance to this role.

AIAWSPythonGCPCloudBackendScalabilitySparkDatabricksNoSQLSQLAirflow

Full listing

Role description

We seek a skilled Lead Python Data Engineer to join our team and help us build and maintain our data pipelines.

As a Python Data Engineer, you will be responsible for designing, developing, and testing Python-based data pipelines and performing data cleansing, transformation and quality assurance.

Responsibilities

  • Design, develop, test and maintain Python-based data pipelines
  • Write high-quality and maintainable Python code to migrate data between different storage systems with modifications
  • Create ETL jobs, airflow operators, and Spark jobs to enhance data processing capabilities
  • Implement Data Intake processes for Data Sources (ETL/ELT, DWH Modelling and Optimization)
  • Perform data cleansing, transformation and quality assurance to maintain data integrity
  • Optimize data processing, storage and retrieval to improve performance
  • Collaborate closely with cross-functional teams to ensure data accuracy and consistency

Requirements

  • In-depth learning of Python programming with the ability to write maintainable and scalable code
  • Strong understanding of distributed computation and multiprocessing
  • Knowledge of data warehouse concepts and ETL processes
  • Proficiency in working state machines, web back-end development, networking and protocols
  • Experience with cloud computing platforms, such as AWS or GCP, and tools like Step Functions, Batch
  • Knowledge of Spark, Hive, Databricks and other data processing frameworks
  • Familiarity with data orchestration tools like Airflow
  • Expertise in with SQL, NoSQL and cloud-based data storage solutions