Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35
We are seeking someone with a strong technical background in building Java-based microservices, AWS Glue, PySpark, SQL, and Athena to join our team.
In this role, you will design, build, and optimise data pipelines and platforms in a cloud environment.
Due to the nature of the role, active SC clearance is required.
Key Responsibilities:
- Data Pipeline Development: Design, develop, and maintain scalable data pipelines using AWS Glue, PySpark, and Java-based microservices to process large volumes of structured and unstructured data.
- Microservices Development: Build and optimize Java-based microservices to support data integration, transformation, and orchestration for real-time and batch data processing.
- AWS Services Management: Leverage AWS services such as S3, Glue, Athena, Lambda, and Step Functions to build and optimize cloud-native data solutions.
- SQL and Data Queries: Write complex SQL queries to analyse data and support business intelligence tools, ensuring efficient querying and data retrieval using AWS Athena and other querying tools.
- Team Leadership: Lead and mentor a team of data engineers, providing technical guidance, setting development standards, and ensuring high performance across the team.
- Client Stakeholder Engagement: Collaborate with key client stakeholders to understand business requirements, translate them into technical solutions, and ensure the successful delivery of data projects.
- Data Architecture: Work with data architects to define data models, design data storage solutions, and optimize data integration processes to ensure data quality, performance, and scalability.
- Data Quality and Governance: Implement data quality controls, monitoring, and governance practices to ensure the integrity and security of data across the platform.
- Performance Tuning: Identify bottlenecks in data processes and optimize performance by fine-tuning Glue jobs, PySpark code, SQL queries, and microservices.
Required Skills and Experience:
- Experience: 6+ years of hands-on experience in data engineering, with a focus on building cloud-based data platforms.
- Java Microservices: Strong expertise in developing and deploying Java-based microservices in a cloud environment.
- AWS Glue & PySpark: Proven experience with AWS Glue and PySpark to build, schedule, and run ETL/ELT processes at scale.
- SQL Expertise: Advanced proficiency in SQL for querying and optimizing data in cloud-based environments like AWS Athena or Redshift.
- Cloud Technologies: Deep understanding of AWS services (S3, Lambda, Glue, Athena, Step Functions) and experience with cloud data infrastructure.
- Team Leadership: Experience leading a team of data engineers, providing mentorship, technical oversight, and fostering collaboration.
- Client Interaction: Strong communication skills with experience working directly with client stakeholders to gather requirements and deliver solutions that align with business objectives.
Preferred Skills:
- Big Data Technologies: Familiarity with big data ecosystems and tools such as Hadoop, Spark, Kafka, or similar.
- Automation & CI/CD: Experience with CI/CD pipelines, containerization (Docker), and orchestration tools (Kubernetes).
- Data Governance: Knowledge of data governance frameworks, data security, and compliance practices in cloud environments.
- Certifications: AWS certifications such as AWS Certified Data Analytics or AWS Certified Solutions Architect are a plus.
Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35