Senior Big Data Engineer Job at Qode, Chapel Hill, NC

Nk1iUGlaV2JNN0dVWlpONmhtWFJqamRXTlE9PQ==
  • Qode
  • Chapel Hill, NC

Job Description

Senior Big Data Engineer Scala/Spark

We are looking for a highly skilled Senior Big Data Engineer with strong expertise in Scala and Apache Spark to support our data engineering team. The ideal candidate should be able to read and refactor existing Scala Spark code, develop enhancements, create robust unit tests, and help migrate code into a modern cloud-native environment.

Key Responsibilities:

  • Analyze and refactor existing Scala Spark code to support migration to new environments.
  • Write and enhance Scala Spark code with a focus on performance and maintainability.
  • Develop comprehensive unit tests to ensure code quality and stability.
  • Work with data stored in Amazon S3 (CSV and Parquet formats).
  • Integrate and manipulate data from MongoDB.
  • Independently set up environments to validate and test code changes.
  • Execute both manual and automated testing procedures.
  • Collaborate within an Agile team and contribute to sprint planning, story grooming, and reviews.
  • Maintain clear and proactive communication with team members, project managers, and stakeholders.

Technical Skills & Qualifications:

  • Proficiency in Scala and Apache Spark for large-scale data processing.
  • Working knowledge of Python for scripting and data transformation.
  • Hands-on experience with Amazon Web Services (AWS), including:
  • S3
  • AWS Glue
  • Step Functions
  • Experience working with MongoDB and structured/unstructured data.
  • Familiarity with SQL and BDD frameworks.
  • Experience with Terraform for infrastructure-as-code is a plus.
  • Strong understanding of software development lifecycle (SDLC), from development to production deployment.
  • Experience using tools like Git, Azure DevOps/TFS, and JIRA.
  • Comfortable working in Agile/Scrum environments.

Core Technologies:

  1. Scala
  2. Spark
  3. AWS Glue
  4. AWS Step Functions
  5. Maven
  6. Terraform

Soft Skills:

  • Strong problem-solving abilities and attention to detail.
  • Ability to work independently and as part of a collaborative Agile team.
  • Excellent written and verbal communication skills.

Job Tags

Similar Jobs

Choctaw Nation

Wildlife Conservation Education and Outreach Technician Job at Choctaw Nation

 ...evident as it continues to focus on providing opportunities for growth and prosperity.Job Purpose or Objective(s): The Wildlife Conservation Education and Outreach Technicianwill educate tribal citizens on conservation issues, provide information to tribal citizens... 

United States Army

Cyber Operations Specialist Job at United States Army

 ...completion of first term contract, you are guaranteed up to 5 interviews with your choice of 1,200 industry leading organizations including COX Communication, Comcast, and Amazon. Similar Career Fields Include: Information Security Analyst, Network Administrator, and... 

El Encanto, A Belmond Hotel

Front Office Assistant Manager (Santa Barbara) Job at El Encanto, A Belmond Hotel

 ...and flexible spending. The Company offers paid time off, sick pay, a 401(k) program with company matching, and an employee assistance program. Team members also enjoy free cafeteria meals and discounts on food and beverage, spa treatments, and retail boutique items... 

Sanford Health

Cardiac Monitor Tech - Part Time | Sanford Health Job at Sanford Health

 ...Join Our Team and Grow The Good! Position: Cardiac Monitor Tech Hours: Rotates between 12 hour Days (6:00am - 6:3...  ...patient responses. Obtains and records patient information and medical histories; interprets electrocardiogram (EKG) results and generates... 

Addison Group

Ocean Freight Forwarder Job at Addison Group

 ...Job Description Job Description Job Title: Ocean Freight Forwarder Location: Erlanger, KY Industry: Logistics Pay: $52,000 - $60,000 per year, depending on experience About Our Client: Our client is a leading logistics organization dedicated to...