Exciting opportunity for a Data Engineer to join a Global Engineering and professional services organisation based in the city. Our client is currently going through a digital transformation programme in response to their recognition of technical changes they need to make in order to remain competitive within their market.
Therefore, our client is currently looking to build out their Data Science and Data Engineering capability.
We are looking for a Data Engineer to come onboard with exposure to Paas, ideally experience with AWS or Azure as we require someone to hit the ground running within a cloud environment. In addition, we are seeking someone with a consultative approach to their work and the drive to want to build upon their current technical ability.
• Deliver the data engineering aspects of projects, services and products
• Work with colleagues to understand client and market requirements, and define how these translate into data engineering workflows
• Work with our Digital Technology Group (DTG) to actively contribute to the development of production environments for data engineering workflows
• Work with colleagues to implement key data infrastructure capabilities including: global availability and multi-jurisdiction data residency, platform multi-tenancy, API management and use of microservices
• Contribute to the development and maintenance of the data catalogue, and associated schemas
• Contribute to the adoption and integration of software development best practices (e.g. agile methods, test-driven development, CI/CD) into data engineering practices
• Be responsible for the validity and quality of data engineering workflows and processes
• A talent for and demonstrable experience of designing scalable, effective, high performance data engineering systems and workflows to support data science, machine-learning, and data-driven design processes
• Fluency in a mainstream object-oriented programming language that can support data engineering tasks, preferably Python
• Confidence working on the command line in a Linux/Unix environment
• Experience working in a cloud environment, preferably AWS
• Experience of using PaaS services from providers such as AWS, e.g. Lambda, EMR, Batch, Kinesis
• Extensive experience in data analytics and stream-processing frameworks, especially from the Apache ecosystem (e.g. Hadoop MapReduce, Spark, Kafka, etc), and their PaaS equivalents
• Data modelling / schema design (relational and NoSQL)
• SQL and NoSQL databases (e.g. Cassandra, MongoDB) and their PaaS equivalents
• A talent for designing effective system architectures