About the Company
This company specializes in data engineering and Big Data solutions, helping organizations integrate, process, and analyze large-scale data. Its focus is on delivering high-performance, scalable data pipelines while collaborating closely with data science and analytics teams. The organization emphasizes innovation, reliability, and advanced technology adoption to drive business insights.
About the Role
The Scala Developer role is focused on developing, optimizing, and maintaining data pipelines using Scala and Spark. The position involves implementing ETL/ELT processes, working with distributed systems, and ensuring high data quality. The developer will collaborate with data science and analytics teams to support transaction monitoring, risk scoring, and other data-driven solutions.
Key Responsibilities
- Develop and optimize data pipelines using Scala and Spark
- Process large volumes of data in batch and near real-time
- Design and implement ETL/ELT workflows using Airflow or Oozie
- Work with distributed systems and optimize their performance
- Integrate data from multiple sources and formats
- Collaborate with data science and analytics teams
- Implement logic for transaction monitoring and risk scoring
- Monitor and validate data quality through profiling and checks
Required Skills
- Strong proficiency in Scala and functional programming
- Experience with Apache Spark
- ETL/ELT pipeline development (Airflow, Oozie)
- Data engineering experience with Parquet, JSON, and batch processing
- Experience with distributed systems
- Hands-on experience with Big Data environments
Nice to Have
- Experience with Quantexa
- Migration experience from Oozie to Airflow
- Knowledge of data governance and monitoring tools (Prometheus, Grafana)
- Experience with graph analytics