Senior Data Engineer
NIRA
Data Science
Bengaluru, Karnataka, India
Posted on Monday, August 28, 2023
We are looking for an experienced Data Engineer to join our engineering team. The hire will be responsible for building our data and data pipeline architecture. You will optimise our data flow starting with the collection of data for cross functional teams and purposes. You will support our key data science initiatives while maintaining consistency of data delivery architecture throughout ongoing projects. Ideal candidates must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of optimising or even re-designing our company's data architecture to support our next generation of products and data initiatives.
Responsibilities
- Create and maintain optimal data pipeline architecture
- Assemble large, complex data sets that meet performance and business requirements
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data' technologies
- Identify, design, and implement internal process improvements: automating manual processes, optimising data delivery, re-designing infrastructure for greater scalability, etc.
- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, credit risk, operational efficiency and other business KPIs.
- Create data tools for analytics and data scientist team members that assist them in building and scaling our core products
- Work with cross domain data and analytics experts to strive for stronger data driven outcomes for the business.
- Advanced SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
- Experience building and optimising big data' data pipelines, architectures and data sets.
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Working knowledge of message queuing, stream processing, and highly scalable big data' data stores.
- We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science.
- Experience with big data tools: Hadoop, Spark / PySpark, Kafka.
- Experience with relational SQL and NoSQL databases.
- Experience with object-oriented/object function scripting languages: Python.
- Expertise in data modelling and buiding data driven systems.
- Well versed with Shell Scripting.
- Hands on experience on various AWS services like EMR, EC2 Glue.
- Deploy existing data projects using CICD pieplines.
- Knowledge on Docker is a plus.
