Home / JOBS / Senior Spark/Scala Developer

Senior Spark/Scala Developer

Toronto
Apply Now

Company Social & Media:

Abstract logo: black head silhouette with a lightbulb as the brain and an orange arc above, symbolizing ideas.

About the Company

Cogency Inc. is a technology organization operating within the banking and financial services sector. The company focuses on supporting enterprise Anti-Money Laundering (AML) initiatives by delivering large-scale data engineering and backend solutions for regulatory, risk, and compliance use cases.

About the Role

A Senior Backend Developer – AML (Big Data) position is available in Toronto, ON, with a hybrid working model requiring three days onsite. The role focuses on building and maintaining scalable backend systems and big data pipelines for high-volume financial data processing within AML platforms. It combines backend development, data engineering, and production support responsibilities in an enterprise environment.

Responsibilities

Big Data Engineering

  • Design, develop, and maintain large-scale data processing applications using Apache Spark and Scala.
  • Build and maintain ETL/ELT pipelines on Hadoop and Cloudera CDP environments.
  • Develop Spark applications using DataFrames, Datasets, and Spark SQL.
  • Optimize Spark workloads through partitioning, caching, broadcast joins, and performance tuning techniques.
  • Support migration of Spark workloads from Spark 2 to Spark 3 on Cloudera CDP.
  • Process structured and semi-structured data using formats such as Parquet, ORC, and Avro.
  • Ensure scalability, performance, reliability, and data quality across enterprise data platforms.

SQL & Data Engineering

  • Develop complex HiveQL and Spark SQL queries using window functions, CTEs, aggregations, and subqueries.
  • Design and maintain Hive managed and external tables.
  • Optimize query performance and troubleshoot slow-running SQL workloads.
  • Manage partitioned datasets and enterprise data models.
  • Support HDFS encryption zones and data governance requirements.

Backend Development & API Integration

  • Design and develop backend services for enterprise data ingestion.
  • Build REST API integrations using Python, curl, and enterprise frameworks.
  • Implement OAuth2 authentication, token management, and API security mechanisms.
  • Develop resilient ingestion pipelines with pagination, retry logic, monitoring, and error handling.
  • Parse and process JSON payloads for downstream analytics and AML processing.
  • Integrate enterprise APIs with Hadoop, Hive, and Spark ecosystems.

Unix & Shell Scripting

  • Develop Bash scripts for automation and operational support.
  • Build orchestration scripts with logging, monitoring, and exception handling.
  • Perform HDFS administration tasks using command-line tools.
  • Manage secure authentication using Kerberos and keytab-based access.
  • Automate file transfers and operational workflows.

DevOps & Scheduling

  • Support workload scheduling using tools such as Ansible Automation Platform, Control-M, and Cron.
  • Develop and maintain Ansible playbooks for automated deployments.
  • Support CI/CD pipelines and release automation processes.
  • Manage secrets, artifact versioning, and environment configurations.
  • Monitor job execution and implement alerting mechanisms.

Collaboration

  • Work closely with Data Engineers, AML Analysts, Architects, DevOps Engineers, QA teams, and business stakeholders.
  • Participate in Agile ceremonies including sprint planning, reviews, and retrospectives.
  • Contribute to architecture discussions and technical design decisions.
  • Produce technical documentation and support knowledge sharing.

AI-Assisted Development

  • Use AI coding assistants such as GitHub Copilot to improve productivity.
  • Support AI-assisted code generation, optimization, documentation, and debugging.
  • Explore Generative AI and LLM-based solutions for engineering efficiency and AML use cases.
  • Contribute to AI-driven data quality and anomaly detection initiatives.

Requirements

  • Bachelor’s or Master’s degree in Computer Science, Software Engineering, Data Engineering, or a related field.
  • At least 7 years of backend development experience, including 5+ years in Big Data engineering.
  • Strong hands-on experience with Apache Spark, Scala, Hadoop, Hive, and HDFS.
  • Proven experience building enterprise ETL/ELT pipelines.
  • Strong SQL and data modeling expertise.
  • Experience with HiveQL, Spark SQL, and query optimization techniques.
  • Strong Unix/Linux administration and Bash scripting skills.
  • Experience integrating REST APIs in enterprise environments.
  • Working knowledge of OAuth2, JSON, and RESTful services.
  • Experience with enterprise scheduling and automation tools.
  • Strong analytical, troubleshooting, and performance tuning abilities.
  • Excellent communication and collaboration skills.

Preferred Qualifications

  • Experience with Cloudera CDP (7.x) and migration from Hortonworks HDP.
  • Knowledge of Kerberos, Vault, and HDFS encryption zones.
  • Experience with CI/CD tools such as GitHub Actions and related automation platforms.
  • Experience integrating Spark with Microsoft SQL Server (JDBC).
  • Python scripting experience for automation tasks.
  • Experience with AI-assisted development tools such as GitHub Copilot.
  • Exposure to Generative AI, LLMs, and AI-driven automation.
  • Experience in Banking and Financial Services (mandatory).
  • Experience in Anti-Money Laundering (AML), Financial Crime, Risk, and Compliance domains.

Please refer to the official website below for a comprehensive job description:

Apply Now