Our client is searching for a talented Engineer to join their dynamic team and play a pivotal role in the development of their cutting-edge Data Lake, you must come from the creative sector. This position presents a unique opportunity to take on the exciting challenges associated with processing and analysing vast streams of data, all while contributing to the strategic decisions that define their global footprint.
- Collaborate closely with a small, agile team to enhance and expand their newly-established Data Lake, processing nearly 1 billion data rows on a daily basis.
- Leverage AWS services and modern data processing technologies, such as Apache Iceberg, Dremio, and more, to construct efficient data pipelines and analytical tools.
- Generate both detailed and summarised insights pertaining to music streaming trends, supporting market strategies and partnerships with diverse Streaming Services.
- Review and approve Pull Requests from team members to ensure code quality and compliance with best practices.
- Independently manage legacy data source maintenance and participate in the development of novel and improved data pipelines for the data lake.
- Demonstrate a proactive, self-starter, and self-motivated approach to drive projects forward.
- Collaborate with cross-functional teams and the broader business to gather requirements and deliver impactful solutions.
The ideal candidate will have:
- Multiple year of experience as a Data Engineer, with a proven track record of constructing robust and scalable data pipelines.
- Profound knowledge of software development best practices and agile methodologies.
- Experience in cloud-native and serverless application development on AWS, including proficiency in technologies like AWS Lambda, AWS Batch, and AWS Step Functions.
- Competency in AWS Boto3/CLI for task automation and AWS resource management.
- Proficiency in Python 3 and a background in Python packaging.
- Strong command of SQL and experience with Apache Airflow for orchestrating data workflows.
- Solid understanding of Apache Parquet and Apache Iceberg file formats.
- Familiarity with CI/CD pipelines, utilising tools like Bitbucket Pipelines and Terraform.
- Knowledge of Docker and Bash/Linux scripting.
- Exceptional problem-solving and communication skills.
- A background or keen interest in the music industry is a valuable bonus.
- This revised job description maintains the key information while presenting it in a more concise and engaging manner.
Handle actively welcomes applicants from under-represented backgrounds - we pride ourselves on attracting the best talent for every opportunity through a commitment to equality, diversity and inclusion.