Data Engineer (Core Platform & Optimization)

About the role

The Mission: Build the High-Performance Refinery At Hilbert’s AI, we shorten "months" to "minutes." To achieve this, our internal data systems must be more than just pipelines—they must be a high-velocity refinery. We are looking for a Core Data Engineer to build the foundation that allows our integration team to move 10x faster. While others focus on connecting the pipes for specific customers, you focus on the integrity, flow, and scalability of the entire system. You will own our "shared brain": the common code, the optimization of our ClickHouse clusters, and the AI-driven engines that turn manual integration into automated ingestion. The Hard Problem: Abstracting the Chaos Our challenge is building a "Generic Growth Engine" that stays performant even as we scale to hundreds of diverse B2C retailers. We don't want to write custom code for every new data source; we want to build a meta-platform where new integrations are defined by configuration and generated by AI. Our Current Hurdles: The Programmable Refinery: You aren't just building pipelines; you are building the infrastructure that our Discovery Agent uses to generate pipelines. You’ll design the configuration schemas that allow AI to scaffold Dagster orchestrations automatically. OLAP at Scale: We rely on Clickhouse for sub-second analytical queries. You’ll be responsible for shard tuning and ensuring our canonical models are structured for maximum performance when queried by AI agents. The "Shared Brain": You will architect the core Python libraries that manage sync logic and automated monitoring, providing the "Source of Truth" that our internal agents learn from. Who You Are (The Profile) The Optimizer: You are the person who finds the bottleneck in a query or a memory leak in a container. You think in terms of complexity classes and system reliability. The Front-Line Veteran: You have empathy for the integration struggle. You understand that a platform is only as good as its usability in the field. Note: Our Core engineers start in a Forward Deployed capacity to earn the "battle scars" necessary to build truly useful abstractions. Software Rigorist: You treat data engineering like software engineering. Modular code, CI/CD, and unit testing for data logic aren't "nice to haves", they are your baseline. The Meta-Engineer: You enjoy building systems that build other systems. You see AI-generated pipelines as an engineering challenge in reproducibility and safety. Dagster Ninja: You don’t just use orchestrators; you push them to their limits. You understand how to build dynamic, asset-based workflows that scale. What You’ll Own Core Engine & Agent Frameworks: Build the shared libraries and configuration schemas that power Hilbert’s data integrations and the Discovery Agent’s output logic. ClickHouse Mastery: Take full ownership of our ClickHouse performance, optimizing storage, query speed, ingestion patterns, and how agents query tables on-demand. Observability Architecture: Design the monitoring and alerting stack that detects data drift or pipeline failures before they impact the customer. R&D Ingestion: Investigate new data sources and sync mechanisms (CDC, streaming, API patterns) to expand our integration capabilities. Path from Integration-to-Core We believe that the best platform tools are built by those who have felt the pain of the user. At Hilbert, we offer an Integration-to-Core pipeline. Phase 1 (The Front Line): You can start as a Forward Deployed Engineer, mastering our stack while solving high-stakes enterprise data problems. Phase 2 (The Architect): Once you've mastered the nuances of the data, you can transition into the Core team to turn those learnings into the foundational code that powers the entire company. Bonus Points Experience building a data orchestration or integration engine from scratch. Deep understanding of E-commerce/Retail data structures (SKUs, Attribution, LTV). Contribution to open-source data tools (Dagster, Airbyte, ClickHouse, etc.). Experience working alongside ML Engineers to optimize data for model training/inference. Having built an agentic workflow before. Location San Francisco, or Istanbul At least 5 hours overlap with PST timezone (7am-5pm) Compensation Competitive salary + equity package, commensurate with experience. Performance-based bonuses tied to project milestones and customer impact. The Hiring Journey Short form → Intro Call → Technical working session → Team conversations → Offer Fast, human, no bureaucracy.