About the Company
The company is looking for a Senior Data Engineer to support the development of scalable data platforms and real-time data processing solutions. The role focuses on building and optimising modern data engineering infrastructure within cloud-based environments.
About the Role
The position involves designing, developing, and maintaining batch and streaming data pipelines using Spark and Scala technologies. The engineer will work with distributed systems, cloud platforms, and modern data tools while collaborating with cross-functional teams to deliver scalable and reliable data solutions.
The role also includes responsibilities related to data governance, system optimisation, and performance improvement within large-scale data environments.
Responsibilities
- Design and develop scalable batch and real-time data pipelines using Spark, Scala, and Spark Streaming
- Build and manage data workflows using Apache Airflow
- Develop and optimise data solutions on Databricks with Delta Lake
- Integrate and manage data across Snowflake and AWS environments
- Work with AWS services including S3
- Ensure data quality, reliability, and pipeline performance
- Collaborate with cross-functional teams to gather and implement data requirements
- Support data governance, privacy, and security best practices, including CCPA and GDPR compliance
- Troubleshoot and optimise performance issues within distributed systems
- Contribute to scalable and high-performance ETL and ELT architectures
Requirements
- 5+ years of experience in Data Engineering or Big Data environments
- Strong hands-on experience with Apache Spark using Scala and Spark Streaming
- Experience with Databricks and Delta Lake
- Experience with Apache Airflow
- Experience working with Snowflake
- Strong knowledge of AWS services, including S3
- Strong programming skills in Java and Scala
- Deep understanding of distributed data processing systems
- Experience building scalable and high-performance data pipelines
- Strong analytical and problem-solving skills
- Good communication and stakeholder management abilities
- Ability to work in fast-paced collaborative environments
Nice to Have
- Experience with Datadog monitoring tools
- Working knowledge of Bash and Python
- Familiarity with SBT (Scala Build Tool)
- Experience with GitHub Actions and CI/CD pipelines
- Understanding of data privacy regulations such as CCPA and GDPR
- Experience with real-time streaming architectures
- Experience with cloud-native data platforms
- Background working with large-scale data platforms