About the role
Role Overview Own and build the full diagnostic, observability, and RCA infrastructure that makes Sage Care’s AI assistant trustworthy and debuggable —in real time and post-call. This engineer builds the visibility layer across telephony, transcription, reasoning, SOP traversal, and tool-calling; creates dashboards for both engineers and live human supervisors; and implements automated triage + notification pipelines that surface issues to the right module owners immediately. This role sits at the intersection of LLM orchestration, voice pipelines, transcription, SOP engines, and operations , serving as the connective tissue across the stack. Your work enables rapid root-cause analysis, real-time intervention, and continuous improvement of our clinical AI assistants. Key
Responsibilities Root Cause Analysis, Tracing & Observability Build automated RCA pipelines to detect and classify failure modes: Hallucinations Misrouted intents Leaked/invalid tool calls (Transfer, SayMessage, Hangup, NOOP) Unrecoverable SOP loops Broken state transitions Telephony dropouts / DTMF issues Implement event tracing infrastructure capturing every agentic decision across LLM, telephony, and SOP execution. Compare expected vs. actual SOP behavior using protocol-driven expectations or human-labeled ground truth. Automatically compute performance, safety, reliability, and coverage metrics. Diagnostic Dashboards & Visualization Build live and post-call dashboards that visualize: Full call timeline SOP/state machine traversal Agent reasoning traces Tool invocation history Divergence from expected behavior Design interactive visualizations: heatmaps, decision-path overlays, branching comparisons, and error hotspots. Build triage dashboards for engineering and operations teams to rapidly understand system health. Integration with Core AI Modules Voice + Telephony Integration Trace call-level events (dropouts, retries, audio playback issues). Detect DTMF misfires and incorrect action routing. Transcriber Module Integration Analyze turn segmentation, word-error-rate drift, boosting performance, and latency. Visualize errors in context (audio, transcript, aligned timecodes). LLM Orchestration Integration Audit intent classification accuracy and subgraph routing. Trace reasoning sequences, missing tool calls, redundant tool calls, or invalid arguments. Validate tool call correctness (maps, SMS, search, internal SOP tools). Live Monitoring & Human-in-the-Loop Architect a live SOP state-machine tracer with: Real-time transcript overlays Current state + next expected state Deviation alerts Build dashboards to monitor 10–15 concurrent calls , highlighting sessions with: Loops Latency spikes Failed tool calls Repeated incorrect decisions Provide human specialists with escalation alerts and context. Command & Control Interface Build an intervention console for on-call specialists, enabling: “Skip step” “Say apology” “Escalate to human” “Send SMS” “Repeat last message” Override of SOP steps while maintaining auditability and continuity . This system must blend seamlessly into existing agent workflows without breaking call integrity. Failure Classification, Clustering & Pattern Detection Build clustering systems (via embeddings or metadata) to group systemic failure modes: Intent misroutes under noisy audio Repeated missing tool calls Looped state machine traversal Hallucinated follow-ups or invalid summaries Generate recurring-failure reports to guide engineering improvements. Auto-Triaging & Notification System (NEW) Design and implement an automated triage and notification system that: Detects failure category and severity in real time. Routes incidents to the correct module owners: Telephony Transcription LLM orchestration SOP/decision-tree team Platform reliability Sends structured payloads containing: Trace graphs Relevant logs Transcript segments SOP divergence snapshots Suggested RCA labels Notifications may integrate with: PagerDuty Slack (rich message blocks) Jira auto-ticket creation Internal incident pipelines This ensures rapid operational feedback loops and reduces time-to-resolution. Post-Call RCA Pipelines & Analytics Extend pipelines to automatically generate human-readable failure summaries with: Call-level trace graphs Tool call sequences Transcript context Classified failure types Suggested root causes Store snapshots for operational handoff and debugging. Required
