Evals Lead

LLM-as-Judge & Evaluation Systems

• Design, build, and maintain LLM-as-judge systems that score production conversations at scale.
• Develop robust evaluation rubrics, judge prompts, calibration methodologies, and reliability testing frameworks.
• Validate evaluator performance against expert human review.
• Establish standards for inter-rater reliability, construct validity, and statistical confidence.
• Continuously improve evaluator accuracy and robustness as products evolve.

Continuous Evaluation Infrastructure

• Build systems that evaluate every production interaction, not just sampled conversations.
• Develop automated evaluation pipelines across voice, text, workflow, and operational signals.
• Partner with Engineering to create scalable evaluation infrastructure across the Confido platform.
• Implement regression detection, benchmark tracking, and release-quality measurement systems.
• Ensure evaluation outputs are actionable, timely, and integrated into day-to-day product development.

Voice & Speech Evaluation

• Build Confido's voice evaluation capabilities across:

• Speech recognition accuracy

• Clinical entity transcription fidelity

• Turn latency

• Endpointing performance

• Barge-in handling

• Voice naturalness

• Multilingual performance

• Evaluate and benchmark voice stack vendors against production traffic.

• Identify whether failures originate from STT, LLM reasoning, TTS, retrieval, orchestration, or workflow logic.

• Develop component-level evaluation methodologies that isolate root causes and guide optimization efforts.

Analytics & Business Impact

• Connect evaluation outcomes directly to business metrics, including:

• Automation rate

• Escalation rate

• Workflow completion

• Client retention

• Revenue impact

• Operational efficiency

• Build reporting systems and dashboards that help teams understand performance and prioritize improvements.

• Partner with customer-facing teams to demonstrate measurable value delivered to clients.

• Establish baselines and prove quality improvements over time through rigorous measurement.

Cross-Functional Leadership

• Partner closely with AI Engineers, Product Managers, Forward Deployed Engineers, and Leadership to drive quality improvements.
• Review evaluation frameworks, rubrics, and methodologies across teams.
• Raise the technical bar for evaluation throughout the organization through mentorship, pairing, and technical review.
• Help establish best practices for experimentation, benchmarking, and product quality measurement.

A meeting lands on your calendar

If they’re a yes, I book the chat. You show up — that’s the whole job-hunt.