Data Engineer
Responsibilities:
- Collaborating with the data science and product teams to execute on product goals.
- Developing and managing fault-tolerant, scalable data pipelines capable of handling terabytes of data using distributed cloud technologies.
- Orchestrating data flows for automated data delivery, while comprehending dependencies and producing data lineage.
- Assisting in the construction of control plane infrastructure using event-driven services for testing, code promotion, and job execution.
- Conducting POCs to validate new tools and services that enhance our data engineering solutions and products.
- Troubleshooting data quality issues and ensuring data integrity.
- Staying abreast of industry standards and technological advancements to continually improve our engineering output.
Requirements:
- Bachelor’s Degree in Computer Science, Engineering, or a related field required. Advanced degree is a plus but not mandatory.
- Proficiency in SQL and at least one programming language, preferably Python.
- Hands-on experience with Apache Spark.
- Familiarity with Google BigQuery
- Preferred experience with Apache Kafka and Spring Boot.
- Familiarity with AWS data management stack (RDS, EC2, S3, etc.).
- Experience in DevOps is a definite advantage.
- Proven ability to efficiently manipulate extremely large datasets.
- Flexibility and agility to adapt to the dynamic demands of a startup environment.