Understudy Labs

An open-source toolkit to capture traces, evaluate cheaper models against benchmarks, and ship specialist routes you own.

Capture traces from LLM production workflows with a single install that deploys within coding agents you already use. Hosted infrastructure is optional.

Evaluate the captured traces and set a benchmark for success. Every future model switch meets or exceeds it in A/B testing.

Train and fine-tune a new model on prompts and weights you always own. Start locally and scale into cloud-hosted processes as you see success.

Deploy a new model only when the held-out eval is beaten. Serve it wherever you want. Production data feeds back into training and compounds performance over time.

Test. Learn. Deploy. Repeat.