About the role
Sets the technical direction for the Copilot offline evaluation platform and partners with teams across the Copilot organization to turn ambiguous quality questions into rigorous, reproducible evaluations—the scenarios, metrics, and scorecards that gate what ships. Leads the architecture of an AI-first, agentic-first evaluation platform in which autonomous agents are first-class operators, designing the services, pipelines, and tooling that agents can operate, extend, and reason about, with the observability and guardrails required to make agent-driven operation trustworthy. Owns the platform's hardest challenge: delivering trustworthy results across a deep chain of dependencies pushed far beyond their normal operating envelope, with designs for graceful degradation, intelligent retries, dependency-aware gating, and reliability under sustained and growing load. Defines how Copilot's agentic experiences are evaluated end to end, from multi-turn trajectories and tool selection to task outcomes and modalities such as UX and voice, building the simulation, scraping, and scoring capabilities needed to make agent behavior measurable. Leads by example and mentors engineers across teams to build extensible, maintainable systems, driving modernization of the evaluation runtime and platform so it can scale with rapidly growing demand while improving cost efficiency and latency. Stays at the forefront of agentic systems, large-scale AI evaluation, and autonomous operations, bringing emerging patterns and practices into the organization and sharing that knowledge to raise the bar for how Copilot measures quality. Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. These
requirements include but are not limited to the following specialized security screenings: Master's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 12+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. Deep experience with large-scale distributed systems and reliability engineering, especially delivering dependable outcomes across many interacting service dependencies operating under heavy or unusual load. Experience building AI-first or agentic systems, and designing systems, APIs, and tooling intended to be operated by autonomous agents, including the observability, guardrails, and safety mechanisms that make agent-driven operation trustworthy. Experience with AI/ML evaluation, measurement, or benchmarking systems, such as LLM-based grading, metric design, or large-scale data and quality pipelines.
