Braintrust (bt)

Run 'bt eval' on the current prompt

Modify prompt in the playground

Re-run eval on the same dataset

Compare results in the side-by-side diff view

Route application traffic through Braintrust Proxy

Set cache TTL and similarity threshold

Monitor 'Cache Hit Rate' in the dashboard

Automate credit alerts

Create a dataset from GPT-4 production logs

Run an evaluation using Llama-3 as the target

Use an LLM-judge to compare accuracy

Identify specific failure modes of the smaller model

Configure a 'Factuality' scorer

Run eval on document extraction tasks

Flag any outputs with low factuality scores for manual human review

Iterate on chunking strategy

Export low-confidence logs to the Braintrust UI

Assign logs to subject matter experts for manual grading

Use manual scores to fine-tune the automated LLM-judge

Set up permanent logging with PII scrubbing

Apply 'Advice Detection' scorers to all production traffic

Generate monthly compliance reports for auditors

Filter production logs by high user-rating scores

Export the high-performing subset as a JSONL dataset

Upload to fine-tuning provider directly from Braintrust