About the role
What we're looking for:
We need an AI Engineer with at least 1-2 years of experience building LLM-powered products in production, plus an additional 1-3 years in a software or ML engineering role. You should be comfortable in a fast-paced, early-stage environment and have a track record of shipping, deploying, and monitoring LLM applications over time. Bonus points if you have experience with recommendation systems or have worked in a consumer-facing product environment.
What you'll do:
- Design and maintain the prompts, structured outputs, and orchestration systems that power Overtone’s AI-driven features.
- Take ownership of building evaluation infrastructure to measure AI quality at scale using tools like Langfuse, TensorZero, and Promptfoo.
- Work closely with product, engineering, and research to bring AI capabilities into real user-facing features with scalability and reliability in mind.
- Run structured experiments across models, prompts, and configurations to optimize quality, cost, and latency.
- Lead the development of internal tools—dashboards, admin interfaces, debugging workflows—that give the team visibility into system behavior and model performance.
- Identify and implement opportunities to fine-tune open-source models using techniques such as SFT, DPO, and RLHF to improve cost and latency.
