APPLYDJINN
Vacancy catalog
EPAM
Open role

Lead Data Software Engineer with Databricks with Apache Kafka, Apache Spark, Kubernetes

EPAMGeorgia; Armenia; Kazakhstan; Kyrgyzstan; Uzbekistan
Work model
Remote
Experience
5+ years
Employment
Not specified
Compensation
Not disclosed
Technology signal
10 tags

Technology context

10

Parsed from the vacancy text; ordered by relevance to this role.

AIJavaPythonKubernetesCloudKafkaSparkDatabricksAnsibleData Software Engineering

Full listing

Role description

We are looking for a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as Data Factory, Databricks, Event Hub, and Cassandra, to a cloud-agnostic platform deployable on-premises.

Responsibilities

  • Implement streaming and batch Spark pipelines running on Kubernetes
  • Lead the implementation of objects within the data generator framework
  • Oversee deployment and testing across local and development environments
  • Drive improvement of the current, actively changing solution
  • Conduct bug investigation and resolution
  • Perform unit testing to ensure solution quality
  • Participate in refinement, planning, and demo sessions
  • Guide and mentor team members throughout the implementation phase

Requirements

  • 5+ years of experience with Apache Spark, Confluent/Apache Kafka, and Kubernetes
  • Proficiency in Python
  • Familiarity with the data engineering domain and cloud-agnostic platform migrations
  • Capability to dive deeper into new technologies in a short time during active implementation phases
  • Ability to work independently with syncs with a team lead
  • Strong communication skills and proactiveness
  • Showcase of being a hands-on team player and technical leader
  • Proficiency in English at a B2+ level

Nice to have

  • Knowledge of Java
  • Familiarity with Kafka Connect, KSQL, and Kafka Streams
  • Skills in Ansible and Argo CD
  • Understanding of TDD