Braintrust

What it does

Braintrust positions itself as an observability and evaluation platform for the whole team, from engineering to product. The homepage describes inspecting every agent trace and tool call, searching across logs, and tracking latency, cost, and quality in real time, supported by Brainstore, a database built for AI data at scale with a query engine called Nitro.

Evaluation features include defining datasets, running experiments against real data, comparing prompts and models side by side, and scoring outputs with LLMs, code, or humans. The homepage and pricing page note versioned datasets, playgrounds, experiments, custom charts and trace views, environments, and a built-in Loop agent that can run evaluations, generate test cases, and iterate on prompts.

SDK and workflow details from Braintrust documentation references include Python and TypeScript SDKs installed as the braintrust package, a bt CLI for eval and sync operations, and a GitHub Action that posts experiment comparisons on pull requests. Deployment is primarily cloud, with hybrid and self-hosted options reserved for Enterprise customers.

Verified capabilities

Quick start

Install the SDK and run a first eval

Install the Python SDK and scorer package from the Braintrust SDK README, define a simple Eval with data, task, and scores, then run it with an API key in the environment.

pip install braintrust autoevals

from autoevals import LevenshteinScorer
from braintrust import Eval

Eval(
    "Say Hi Bot",
    data=lambda: [{"input": "Foo", "expected": "Hi Foo"}],
    task=lambda input: "Hi " + input,
    scores=[LevenshteinScorer],
)

Source: https://www.braintrust.dev/docs. Examples use placeholders only. Never paste a real key into a profile, config file you share, or a ticket.

MCP support: Braintrust documentation references state the vendor also publishes an MCP server to query projects, experiments, and logs, run BTQL, and summarize experiments, with a recommendation to prefer the bt CLI for most work.

Works with

Only sourced support is listed. A missing framework means AgentsUse has not verified it yet, not that it cannot work.

Health and maintenance

Pricing and license

Freemium model with a free Starter tier including included processed data and scores with limited retention, a paid Pro tier priced per month with larger included data and score allowances and per-unit overage for additional data and scores, and custom Enterprise pricing. The pricing page meters processed data in GB and scored outputs from LLM-as-a-judge, autoevals, or custom code scorers.

Official pricing or docs →

Limitations and safety

• Self-hosting and hybrid deployment require an Enterprise agreement, which may not fit teams with strict data residency needs on a budget.
• Pricing meters scored outputs, so adding more scorers or scoring every production trace increases cost in addition to data processing fees.
• The workflow is opinionated around connecting production traces to datasets and experiments, which can constrain teams with fully custom evaluation pipelines.

Alternatives to Braintrust

Related tools

Common questions