We are looking for a Junior Data Engineer to design, develop, and optimize our data infrastructure on Databricks. You will be involved in architecting pipelines using BigQuery, Google Cloud Storage, Apache Airflow, dbt, Dataflow, and Pub/Sub, ensuring high availability and performance across our ETL/ELT processes.
A successful candidate has knowledge of cloud-native data solutions, experience with ETL/ELT frameworks, and a passion for building robust, cost-effective pipelines.
Key Responsibilities
Data Architecture
- Support the development and maintenance of our data platform on GCP, including data warehousing in BigQuery/Databricks and data lake storage in Google Cloud Storage.
- Help organize data into clear layers and domain-focused Data Marts for analytics and reporting.
- Assist with Terraform-based Infrastructure as Code to provision and manage cloud resources in a consistent way.
- Contribute to batch and near real-time data workflows with a focus on reliability, scalability, and cost awareness.
Pipeline Development & Orchestration
- Build, maintain, and improve ETL/ELT pipelines under guidance using Apache Airflow for workflow orchestration.
- Develop and maintain dbt transformations to create clean, version-controlled data models in BigQuery.
- Support data ingestion and processing using tools such as Google Dataflow, Apache Beam, or Pub/Sub where needed.
- Monitor scheduled jobs, troubleshoot failures, and help ensure data is delivered on time for analytics and reporting.
Data Quality, Governance & Security
- Help implement and maintain data quality checks using Great Expectations, dbt tests, or similar tools.
- Support documentation of datasets, metadata, lineage, and audit processes.
- Follow security best practices, including IAM, encryption, and secure handling of sensitive data.
- Assist in maintaining compliance with data privacy and governance requirements such as GDPR or CCPA.
Scientists & Analytics Enablement
- Partner with Analytics, Product, and Data Science teams to provide reliable datasets for dashboards, reporting, and experimentation.
- Help maintain Data Marts that support key business domains and stakeholder needs.
- Support data availability and accessibility for analytics and machine learning use cases.
- Learn from senior team members and grow into owning larger parts of the data platform over time.