Data Engineer
NIRA
Software Engineering, Data Science
Bengaluru, Karnataka, India
Posted on Dec 28, 2024
We are looking for an experienced Data Engineer to join our engineering team. The hire will be responsible for building our data and data pipeline architecture. You will optimize our data flow starting with the collection of data for cross-functional teams and purposes. You will support our key data science initiatives while maintaining consistency of data delivery architecture throughout ongoing projects. Ideal candidates must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products. The right candidate will be excited by the prospect of optimizing or even re-designing our company's data architecture to support our next generation of products and data initiatives.
Responsibilities
Responsibilities
- Create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet performance and business requirements.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data technologies.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, credit risk, operational efficiency, and other business KPIs.
- Create data tools for analytics and data scientist team members that assist them in building and scaling our core products.
- Work with cross-domain data and analytics experts to strive for stronger data-driven outcomes for the business.
- Advanced SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
- Experience building and optimizing big data data pipelines, architectures, and data sets.
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Working knowledge of message queuing, stream processing, and highly scalable big data data stores.
- We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science. T
- Experience with big data tools: Hadoop, Spark / PySpark, Kafka.
- Experience with relational SQL and NoSQL databases.
- Experience with object-oriented/object function scripting languages: Python.
- Expertise in data modeling and building data-driven systems.
- Well-versed with Shell Scripting.
- Hands-on experience on various AWS services like EMR, and EC2 Glue.
- Deploy existing data projects using CICD pipelines.
- Knowledge of Docker is a plus.
