Our client is searching for a talented Engineer to join their dynamic team and play a pivotal role in the development of their cutting-edge Data Lake, you must come from the creative sector. This position presents a unique opportunity to take on the exciting challenges associated with processing and analysing vast streams of data, all while contributing to the strategic decisions that define their global footprint.
Key Responsibilities:
- Collaborate closely with a small, agile team to enhance and expand their newly-established Data Lake, processing nearly 1 billion data rows on a daily basis.
- Leverage AWS services and modern data processing technologies, such as Apache Iceberg, Dremio, and more, to construct efficient data pipelines and analytical tools.
- Generate both detailed and summarised insights pertaining to music streaming trends, supporting market strategies and partnerships with diverse Streaming Services.
- Review and approve Pull Requests from team members to ensure code quality and compliance with best practices.
- Independently manage legacy data source maintenance and participate in the development of novel and improved data pipelines for the data lake.
- Demonstrate a proactive, self-starter, and self-motivated approach to drive projects forward.
- Collaborate with cross-functional teams and the broader business to gather requirements and deliver impactful solutions.
The ideal candidate will have:
- Multiple year of experience as a Data Engineer, with a proven track record of constructing robust and scalable data pipelines.
- Profound knowledge of software development best practices and agile methodologies.
- Experience in cloud-native and serverless application development on AWS, including proficiency in technologies like AWS Lambda, AWS Batch, and AWS Step Functions.
- Competency in AWS Boto3/CLI for task automation and AWS resource management.
- Proficiency in Python 3 and a background in Python packaging.
- Strong command of SQL and experience with Apache Airflow for orchestrating data workflows.
- Solid understanding of Apache Parquet and Apache Iceberg file formats.
- Familiarity with CI/CD pipelines, utilising tools like Bitbucket Pipelines and Terraform.
- Knowledge of Docker and Bash/Linux scripting.
- Exceptional problem-solving and communication skills.
- A background or keen interest in the music industry is a valuable bonus.
- This revised job description maintains the key information while presenting it in a more concise and engaging manner.
Handle actively welcomes applicants from under-represented backgrounds - we pride ourselves on attracting the best talent for every opportunity through a commitment to equality, diversity and inclusion.
