Skip to main content
Harbor is a Python framework for running AI agent evaluations in isolated Docker containers. Braintrust’s native Harbor job plugin syncs evaluation results (datasets, experiments, scores, and agent traces) to Braintrust after each Harbor run.

​
Setup

Install Braintrust alongside Harbor. Requires harbor>=0.16.0 and Python 3.12 or later. Docker must be running for Harbor to execute tasks.
1

Install packages

2

Set environment variables

.env

​
Run evaluations

Pass --plugin braintrust to harbor run to enable the Braintrust plugin. The plugin is discovered automatically through Harbor’s braintrust entry point after installation.
The Harbor integration is activated through the --plugin braintrust flag in the harbor run CLI, not through auto_instrument(). No import or patching is required in your task code.
After the run completes, the plugin creates a sync manifest at jobs/my-eval/braintrust-sync.json.

​
Configure with environment variables

Set HARBOR_BRAINTRUST_PROJECT to avoid repeating --plugin-kwarg project_name=... on every run:
.env
Then run without the project_name kwarg:

​
Backfill a past job

To re-sync a completed Harbor job without re-running the evaluation, use backfill_job:
backfill.py
Backfill uses deterministic dataset and experiment identities, so it reconciles with existing Braintrust data instead of creating duplicates.

​
What Braintrust receives

After each Harbor evaluation run, Braintrust receives:
  • A managed dataset with the task cases used in the evaluation.
  • An experiment row for each trial, with the agent’s final output as the experiment output and the Harbor task’s reference answer as the expected output.
  • Scores derived from Harbor verifier rewards, mapped to Braintrust score fields.
  • Agent traces (when trajectory tracing is enabled), showing the agent’s tool calls and model turns using Harbor’s ATIF format.

​
Resources