Lead Data Engineer - SC Cleared - Outside IR35 - 6 Months

  • London

JOB DESCRIPTION

Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35

We are seeking someone with a strong technical background in building Java-based microservices, AWS Glue, PySpark, SQL, and Athena to join our team.
In this role, you will design, build, and optimise data pipelines and platforms in a cloud environment.

Due to the nature of the role, active SC clearance is required.

Key Responsibilities:

  • Data Pipeline Development: Design, develop, and maintain scalable data pipelines using AWS Glue, PySpark, and Java-based microservices to process large volumes of structured and unstructured data.
  • Microservices Development: Build and optimize Java-based microservices to support data integration, transformation, and orchestration for real-time and batch data processing.
  • AWS Services Management: Leverage AWS services such as S3, Glue, Athena, Lambda, and Step Functions to build and optimize cloud-native data solutions.
  • SQL and Data Queries: Write complex SQL queries to analyse data and support business intelligence tools, ensuring efficient querying and data retrieval using AWS Athena and other querying tools.
  • Team Leadership: Lead and mentor a team of data engineers, providing technical guidance, setting development standards, and ensuring high performance across the team.
  • Client Stakeholder Engagement: Collaborate with key client stakeholders to understand business requirements, translate them into technical solutions, and ensure the successful delivery of data projects.
  • Data Architecture: Work with data architects to define data models, design data storage solutions, and optimize data integration processes to ensure data quality, performance, and scalability.
  • Data Quality and Governance: Implement data quality controls, monitoring, and governance practices to ensure the integrity and security of data across the platform.
  • Performance Tuning: Identify bottlenecks in data processes and optimize performance by fine-tuning Glue jobs, PySpark code, SQL queries, and microservices.

Required Skills and Experience:

  • Experience: 6+ years of hands-on experience in data engineering, with a focus on building cloud-based data platforms.
  • Java Microservices: Strong expertise in developing and deploying Java-based microservices in a cloud environment.
  • AWS Glue & PySpark: Proven experience with AWS Glue and PySpark to build, schedule, and run ETL/ELT processes at scale.
  • SQL Expertise: Advanced proficiency in SQL for querying and optimizing data in cloud-based environments like AWS Athena or Redshift.
  • Cloud Technologies: Deep understanding of AWS services (S3, Lambda, Glue, Athena, Step Functions) and experience with cloud data infrastructure.
  • Team Leadership: Experience leading a team of data engineers, providing mentorship, technical oversight, and fostering collaboration.
  • Client Interaction: Strong communication skills with experience working directly with client stakeholders to gather requirements and deliver solutions that align with business objectives.

Preferred Skills:

  • Big Data Technologies: Familiarity with big data ecosystems and tools such as Hadoop, Spark, Kafka, or similar.
  • Automation & CI/CD: Experience with CI/CD pipelines, containerization (Docker), and orchestration tools (Kubernetes).
  • Data Governance: Knowledge of data governance frameworks, data security, and compliance practices in cloud environments.
  • Certifications: AWS certifications such as AWS Certified Data Analytics or AWS Certified Solutions Architect are a plus.

Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35

FIND A NEW OPPORTUNITY.

Software Engineer - React

  • Bath, Somerset
  • Frontend Engineer - ReactContract: Outside IR35Length: 3 Months (Exten...
View Job

Three.js 3D Modelling Software Engineer (Contract)

  • Bath, Somerset
  • Job Title: Three.js 3D Modelling Software Engineer (Contract)Location:...
View Job

Product Owner - Clinical Data Solutions £70,000

  • Bristol
  • SR2 is partnering with an exciting client of ours that develops cuttin...
View Job

Senior Software Engineer

  • Cardiff
  • Senior Software Engineer | Cardiff two days a week | £60-75k Do you wa...
View Job