About the role
The Role
We are looking for a foundational member of our engineering team: a highly motivated Software Engineer to own the design, creation, and evolution of our data platform. You will be part of the team that owns the data ingestion and management infrastructure that powers Crustdata’s capabilities.
If you are passionate about building robust, scalable data systems and want to see your work directly influence customers, this is the role for you.
What You'll Do
- Architect & Build: Design, build, and maintain our core data infrastructure, including our data warehouse and data lake, using modern cloud technologies (AWS, GCP, or Azure).
- Pipeline Development: Develop and scale robust, fault-tolerant data pipelines (ETL/ELT) to ingest and process massive volumes of structured and unstructured data from diverse sources.
- Enable Data Science & ML: Create the foundational platform to support our data scientists and ML engineers. This includes building systems for feature engineering, model training, and deploying ML models into production.
- Orchestration at Scale: Implement and manage workflow orchestration for hundreds of daily data jobs, ensuring reliability, monitorability, and efficiency using tools like Airflow, Dagster, or Prefect.
- Real-time Infrastructure: Build and manage real-time data streaming pipelines using technologies like Kafka or Flink to power live dashboards and time-sensitive product features.
- Data Quality & Governance: Champion data quality and reliability. Implement frameworks for data validation, testing, and monitoring to ensure our data is accurate and trustworthy.
