APPLYDJINN
Vacancy catalog
SoftServe
Open role

Senior Big Data Engineer (Python+Apache Iceberg+AWS experience))

SoftServeUkraine; Poland; Romania
Work model
Hybrid
Experience
5+ years
Employment
Full Time
Compensation
Not disclosed
Technology signal
11 tags

Technology context

11

Parsed from the vacancy text; ordered by relevance to this role.

AIAWSDistributed SystemsPythonReliability EngineeringC++DockerScalabilitySparkLinuxCI/CD

Full listing

Role description

About the role

In this role, you will contribute to large-scale data pipeline migrations, moving trading venue data from existing HDF5-based pipelines to modern Parquet and Apache Iceberg-based storage, while ensuring uninterrupted delivery to customers and downstream systems.

Our Client is a leading provider of harmonised historical market data and analytics, supporting trading and research decisions.

You will work within a focused team of Python and C++ engineers, modernising data parsing pipelines and infrastructure to improve performance, reliability, and scalability.

Responsibilities

  • Develop robust Python code for Iceberg-based data storage, retrieval, migration, and validation, working closely with C++ and Python engineers.
  • Migrate existing parsing processes to the new Parquet and Iceberg-based pipeline, focusing on simplicity, performance, and reliability.
  • Collaborate with business stakeholders and engineering teams to gather requirements, understand constraints, design technical solutions, and validate outcomes.
  • Build, test, and deliver production-ready functionality to support business requirements and ongoing pipeline improvements.
  • Contribute to the development and optimisation of existing parsing pipelines and C++ parsers (feed handlers).
  • Leverage AI coding tools to accelerate development while maintaining responsibility for code quality, consistency, and maintainability.
  • Understand end-to-end data flows, including data ingestion, processing, storage, and delivery to customers.
  • Communicate with internal technical teams and external data providers, including financial exchanges, to support data processing and operational requirements.
  • Provide second-line support for daily parsing processes and data-related issues, working alongside the wider development team.

Requirements

Essential

  • Professional experience in Python development, including writing reliable and maintainable production code.
  • Hands-on experience with Apache Iceberg and pyiceberg libraries.
  • Experience with AWS services, particularly S3 storage, and ideally AWS Glue Data Catalog.
  • Experience with system migrations, ensuring existing end-to-end functionality is maintained.
  • Experience working with large datasets and data-intensive applications.
  • Professional experience developing software in Linux environments.
  • Familiarity with industry-standard development practices, including version control, unit testing, and continuous integration.
  • Strong foundational software engineering skills, with the ability to develop correct and maintainable code independently and with AI assistance.
  • Ability to self-organise and deliver development tasks over projects lasting several months.
  • Strong problem-solving and communication skills.
  • Computer Science or another STEM-related degree.

Nice to Have

  • Experience with Apache Spark.
  • Experience working with large-scale distributed systems.
  • Experience delivering well-tested, production-level code as part of a larger codebase.
  • Exposure to equities or derivatives market data from financial exchanges, including protocols such as ITCH, GTP, EOBI, iMpact, Pillar, or Optiq.
  • Knowledge of containerisation technologies such as Docker.