Braintrust is pivotal to Zapier’s AI strategy, offering unmatched observability, evaluation, and model testing capabilities. From model comparisons to regression analysis, Braintrust equips Zapier to refine and optimize AI-driven solutions effectively.

Braintrust was co-founded by Figma and Zapier, with Figma's AI team lead, Anker, spearheading development. Braintrust started as an evaluation, observability, and prompt-engineering tool with Zapier as its first user.

Zapier integrates Braintrust as a full-stack AI platform for tools like AI Zap Builder and Zapier Co-pilot. Key uses include:

• Evaluation Suite: Zapier leverages Braintrust to run synthetic data evaluations in continuous integration (CI) cycles.
• Custom Graders: Uses both logic-based and LLM-based graders to assess AI outputs.
• Observability: Tracks co-pilot’s performance with granular insights into inputs, outputs, and processing times.
• Model Comparison: Compares performance across models (e.g., GPT-3.5 vs. GPT-4), enabling data-driven model choices.
• Regression Analysis: Detects and addresses performance regressions with comparative analysis.

Zapier’s Braintrust workflow includes loading synthetic data, running CI-based evaluations, applying custom graders, generating reports, and tracing operations for optimization. Braintrust’s observability features allow Zapier to diagnose bottlenecks, compare models, and take corrective action on regressions.

Braintrust.pdf

https://www.youtube.com/watch?v=fOtTHWeU6B8