Lead Data Test Automation Engineer

About the role

In this role, you will lead data quality and testing efforts across large-scale Big Data platforms, ensuring the reliability of data pipelines, processing workflows, analytics, and ML-related solutions. You will work closely with Data Engineering, Data Science, Analytics, and client teams to drive data validation strategies, investigate data issues, mitigate quality risks, and improve the overall quality of the data ecosystem using modern cloud and data technologies.

Responsibilities

• Define test strategies and testing approaches, and coordinate the testing scope across Data Engineering, Data Science, Analytics, and QA
• Identify dependencies, prerequisites, and potential blockers for data test automation solutions
• Manage data quality incidents, communicate findings and risks to clients, and drive issues toward quicker resolution to help unblock the team
• Execute and monitor Airflow DAG runs and validate pipeline results
• Analyze critical datasets and data quality rules, and implement or maintain appropriate checks using Alation or similar tools
• Validate data processing results and key metrics after pipeline execution
• Verify Tableau reports and other Web UI results after data is published
• Support the definition and implementation of regression testing and data validation approaches
• Contribute to improving data quality standards, testing processes, and automation practices

Requirements

• Strong professional experience in Data Testing, with a solid understanding of ETL/ELT and data pipeline testing using Apache Airflow or similar orchestration tools
• Experience testing data-intensive applications and Big Data solutions, AWS experience is preferred
• Advanced SQL skills and strong ability to investigate and validate data using complex queries, Snowflake experience is preferred
• Good Python knowledge and experience creating data validation or automation solutions using Python, Pandas, and/or PySpark
• Experience with data quality and observability tools such as Alation, Soda, or similar solutions
• Strong understanding of data quality dimensions and common data validation techniques
• Experience with regression testing and test automation principles
• Experience investigating, documenting, and resolving data quality incidents
• Understanding of CI/CD and version control practices, such as GitHub Actions or similar tools
• Strong ownership and ability to work independently on complex data quality problems
• Strong communication skills and confidence working with engineering, analytics, and customer teams