Senior Big Data Engineer Job at Qode, Chapel Hill, NC

Nk1iUGlaV2JNN0dVWlpONmhtWFJqamRXTlE9PQ==
  • Qode
  • Chapel Hill, NC

Job Description

Senior Big Data Engineer Scala/Spark

We are looking for a highly skilled Senior Big Data Engineer with strong expertise in Scala and Apache Spark to support our data engineering team. The ideal candidate should be able to read and refactor existing Scala Spark code, develop enhancements, create robust unit tests, and help migrate code into a modern cloud-native environment.

Key Responsibilities:

  • Analyze and refactor existing Scala Spark code to support migration to new environments.
  • Write and enhance Scala Spark code with a focus on performance and maintainability.
  • Develop comprehensive unit tests to ensure code quality and stability.
  • Work with data stored in Amazon S3 (CSV and Parquet formats).
  • Integrate and manipulate data from MongoDB.
  • Independently set up environments to validate and test code changes.
  • Execute both manual and automated testing procedures.
  • Collaborate within an Agile team and contribute to sprint planning, story grooming, and reviews.
  • Maintain clear and proactive communication with team members, project managers, and stakeholders.

Technical Skills & Qualifications:

  • Proficiency in Scala and Apache Spark for large-scale data processing.
  • Working knowledge of Python for scripting and data transformation.
  • Hands-on experience with Amazon Web Services (AWS), including:
  • S3
  • AWS Glue
  • Step Functions
  • Experience working with MongoDB and structured/unstructured data.
  • Familiarity with SQL and BDD frameworks.
  • Experience with Terraform for infrastructure-as-code is a plus.
  • Strong understanding of software development lifecycle (SDLC), from development to production deployment.
  • Experience using tools like Git, Azure DevOps/TFS, and JIRA.
  • Comfortable working in Agile/Scrum environments.

Core Technologies:

  1. Scala
  2. Spark
  3. AWS Glue
  4. AWS Step Functions
  5. Maven
  6. Terraform

Soft Skills:

  • Strong problem-solving abilities and attention to detail.
  • Ability to work independently and as part of a collaborative Agile team.
  • Excellent written and verbal communication skills.

Job Tags

Similar Jobs

PwC

Mergers and Acquisition Partnership Tax - Senior Associate Job at PwC

 ...strategy. These individuals help businesses navigate complex tax...  ...lead and deliver value at this level include but are not limited to...  ...Management, Economics, Finance, International Business ~2 year(s) of...  ...intend to hire experienced or entry level job seekers who will need... 

EPIP

Executive Director (SVPS) (Seattle) Job at EPIP

 ...Venture Partners Seattle is seeking an experienced and versatile Executive Director who can inspire and support people as they work together to...  ...the planned exit of the founding Executive Director. An Interim Executive Director was engaged by the Board in April 2019 to... 

Reveation Labs

$15 - $20 per hour - Hiring Now - Remote Data Entry Operator (Full-Time) Job at Reveation Labs

 ...data and identifying any discrepancies Skills And Experience The following experience is essential to...  ...skills and attention to detail Must have excellent typing speed over 50 WPM Employment Type: Full-Time Salary: $ 15.00 20.00 Per Hour

TIAA

529 Education Savings Consultant Job at TIAA

Join to apply for the 529 Education Savings Consultant role at TIAA3 days ago Be among the first 25 applicantsJoin to apply for the 529 Education...  ...about the 529 College Savings Plans we manage. While working under moderate supervision as a specialist in 529 plans, this... 

Feiyu Technology International Co., Ltd

Overseas Operations Manager Job at Feiyu Technology International Co., Ltd

1. Bachelor's degree or above, with at least 5 years of experience in overseas mobile game publishing, expertise in at least one area of production operations, market promotion, or user acquisition;2. Having strong analytical and planning abilities, with a pioneering...