Senior Big Data Engineer

The Department for Work and Pensions

Leeds, West Yorkshire

Newcastle upon Tyne, UK

£48,806 to £60,535


Before you apply - don't miss out!

Subscribe to our weekly job alert

By registering you agree to our terms & conditions and privacy policy.

You can unsubscribe anytime using the link in the bottom of the email.

You will receive an email asking you to confirm your subscription

Do you want to be part of something really big?

Do you have expertise in Hadoop stack, Spark, Hive or Impala?

Do you have experience in ETL tooling pipelines?

Are you ready to be part of an epic digital transformation?

Then look no further, we'd love to hear from you!

If you're creative and curious to learn, want to be empowered to make decisions and challenge the norm, and want to bring BIG ideas to a MASSIVE digital challenge then you'll love it here. And we'd love to hear from you.

We're looking for talented Big Data Engineers to join our growing community of experts on our Data & Analytics practice and become a key part of one of the biggest and highest-profile digital transformation programmes in the world.

Using fresh ideas and leading edge technologies - and putting the user at the centre of everything we do - we create innovative digital solutions that make a difference to the lives of our 22 million users.

You and your role

The Department for Work and Pensions (DWP) is one of the largest Government Departments, with an enormously ambitious and challenging agenda, often working in partnership with the private and voluntary sector. We provide services and support to over 20 million people, are responsible for over 730 million benefit payments each year, and manage over £180 billion per year. At the same time, we are leading a once in a generation programme of welfare reform to: * Understand and tackle the root causes of poverty rather than its symptoms * Encourage people to work and making work pay * Encourage disabled people and those with ill health to work and be independent * Provide a decent income for people of pension age and promoting saving for retirement * Provide value for money and reducing levels of fraud and error * Reduce work-related death and serious injury in workplaces through the health and safety executive At the heart of this massive agenda is the digital transformation of the services the DWP provides, which will radically improve how government interacts with people. The services will be easier and swifter to access; the time taken to develop new services will be decreased and the cost of delivery reduced significantly. For further information please go to:

The Senior Big Data Engineer role sits right at the heart of that transformation.

Big Data Engineers are responsible for the creation and maintenance of analytics infrastructure, providing the foundations for data needs. You will be responsible for the development, construction, maintenance and testing of architectures, such as databases and large-scale processing systems, implementing data flows to connect operational systems and writing and executing ETL scripts and codes As a Senior Data Engineer you will be responsible for leading, managing and delivering all aspects of data development activities to meet customer needs to agreed timescales

  • You will own the overall Data Engineering Development process within Data Engineering, alongside setting strategies for products and the wider community.
  • You will also provide technical leadership to a team of Data Engineers, along with providing oversight to the products they develop.

This is a big challenge, but also a big opportunity. You will get the chance to work on one of the biggest, most ambitious Digital transformation programmes in the UK. You will have the chance to use and stretch your abilities, strengthen your experience and continue your professional development in a supportive environment. You will work alongside people who care passionately about public service, making a difference and delivering for our customers.

Essential Technical Skills

We'll assess you against these technical skills during the selection process:

  • Technical Breadth
  • Communicating, Analysis & Insight

Candidates should be able to demonstrate in depth knowledge and experience in the application of the following skills; * Extensive proven experience of big data engineering techniques and concepts using the Hadoop Stack (Cloudera/EMR), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala * Extensive, hands-on experience of large complex Data Engineering projects designing and developing ETL pipelines in a cloud or on premise environment * Experience of design and implementation of data storage, including HDFS, S3, relational and NoSQL * Experience of developing/utilising programming and query languages e.g. SQL, Java, Scala

  • Creating Technical requirements
  • Process Modelling
  • Experience of at least one programming language such as SQL
  • Understanding of Data Engineer concepts - creating technical requirements using conceptual, logical and physical designs
  • Project Management methodologies

What will I be doing?

As a Senior Big Data Engineer expect to be.. * Responsible for leading, managing and delivering all aspects of development activities to meet customer needs * Lead the shaping of solutions, ensuring strategic fit, following the agreed principles and contributing to the continuous improvement of the Data Engineer Development lifecycle * Develop, own, maintain and ensure adherence to standards across the Data Engineering community * Promote the development of reusable, robust and scalable products e.g. code, citizen centric data models and exploitation products * Quality assuring documentation for projects, including project plans, and resource estimates * Designing and developing reusable metadata libraries and promoting reuse across the department * Work with other technologists, including Lead Data Engineers and Data Architects to investigate and develop solutions * Develop, maintain and continually improve data engineering, automation and scaling capabilities * Provide technical leadership to individuals and build the Data Engineer community and capability * Responsible for developing complex data processing pipelines in a cloud or on premise environment in line with build standards and ensure solutions are robust and scalable * Provide technical leadership, setting strategic direction to ensure the products that are developed conform to data standards and strategic principals * Responsible for leading, managing and delivering all aspects of code development activities for data ingest routines for unstructured data * Responsible for the implementation and maintenance of coding and tooling standards and for ensuring adherence from the wider community


Before you apply - don't miss out!

Subscribe to our weekly job alert

By registering you agree to our terms & conditions and privacy policy.

You can unsubscribe anytime using the link in the bottom of the email.

You will receive an email asking you to confirm your subscription

Not what you're looking for?

Use our search to find your ideal career opportunity

Search term is required
No jobs were found
Total of [[ totalJobs ]] jobs were found

[[ ]]

[[ jobs.location ]]
[[ jobs.salary_min|decimals ]] - [[ jobs.salary_max|decimals ]]
[[ jobs.posted_time_friendly ]]

Featured Jobs

Audit Scotland

Edinburgh, Scotland

January 12, 2020

Lambeth Council

London, UK

January 03, 2020


Wick Lane, London

December 26, 2019

The Stars Group

Dublin, Ireland

December 26, 2019

Fox School of Business

Philadelphia, PA, USA

December 22, 2019


London, UK

December 25, 2019

NHS Business Services Authority

Newcastle upon Tyne, UK

January 01, 2020


Darmstadt, Germany

December 19, 2019

Mimica Automation

Clifton Street, London

December 21, 2019

University of Hertfordshire

Hatfield, Hertfordshire, UK

December 20, 2019

The Office for Budget Responsibility (OBR)

London, UK

December 25, 2019

The Department for Work and Pensions

Leeds, West Yorkshire

December 14, 2019

Our Partners

Logo for Dstl
Logo for Prifysgol
Logo for Bank Of England
Logo for Cma
Logo for Frontier
Logo for Logo
Logo for Mit Logo

Like what you see?

Post a job