The self-hosted platform that makes prompt engineering far easier
Across the full lifecycle, with automatic, data-driven optimization built in.
Version, regression-test, experiment, optimize, release, and roll back — on data and models you own.
proofhound.org · Reserve a ProofHound Cloud seat
Quick start · How it works · Discord
ProofHound turns prompt engineering into a data-driven, traceable workflow. Instead of stitching together scripts, ad-hoc experiments, spreadsheets, and hand-rolled release logic, you run the whole loop — dataset regression, experiments, automatic optimization, canary and production releases, immutable run results, and rollback — in one place, self-hosted on infrastructure you control. It is built for developers first: clone, `pnpm dev`, connect a model, and start experimenting in minutes. And because the tuning loop is productized around datasets, metrics, and prompt versions, non-engineering teammates can define goals, launch optimizations, and ship releases too. The open-source edition runs as a single-workspace local admin console and keeps a `project_id` data boundary, so it can later connect to an external control plane without changing core resource semantics. ## What you get ProofHound runs one lifecycle, and every stage writes to the same fact tables — so a model invocation stays traceable from a dataset sample all the way to production and back: - **Prompt versions** — every edit creates an immutable version with its variables, output fields, and judgment rules; once an experiment, optimization, or release references it, the version is frozen, so every result maps back to the exact prompt used. - **Dataset regression** — run a version against a dataset of expected outputs (CSV / TSV / JSONL / JSON / ZIP) and get accuracy, precision, recall, F1, per-class metrics, failed samples, and full invocation details — not an aggregate score that hides minority-class behavior. - **Experiments** — batch prompt version × dataset × model, then stop, resume, compare across runs, and export; every run is reproducible because the version it used is frozen. - **Automatic optimization** — analyze failed samples, generate new candidate versions, and re-run regression round by round, targeting class-level goals (e.g. lift recall on a high-risk class) and falling back to the best version when a round regresses. - **Canary & production releases** — promote a proven version through queue-connector canaries with traffic split and dual-run, then 100% promotion, config updates, rollback, and forced stop; webhook ingress can go straight to production. - **Run results** — every stage writes one immutable record per call: input variables, rendered prompt, raw output, structured output, judgment, latency, tokens, and cost. - **Human annotations** — written to a separate table, never mutating the original run results. - **Connectors** — wire prompts to queue connectors and webhook ingress for online traffic. - **MCP channel** — built in, so agents can manage prompt versions, start experiments / optimizations, and query results. - **Bring your own models** — OpenAI, Azure OpenAI, Anthropic, DeepSeek, and more, with your own keys and pricing. ## Get started You need: - Node.js 24 - pnpm - Docker and Docker Compose PostgreSQL, Redis, and the other local dependency services are started for you by Docker Compose — you do not need to install them manually. ```bash git clone https://github.com/proofhound/proofhound.git cd proofhound pnpm install cp .env.example .env pnpm dev ``` `pnpm dev` starts the local dependency services, runs database migrations, and launches server, webhook, worker, and web together. `cp .env.example .env` ships working local defaults, but replace the model API key encryption secret before any non-local run — see [Configuration](#configuration). Default local services: | Service | Address | | ---------------- | -------------- | | Web UI | localhost:3000 | | Server API | localhost:4000 | | PostgreSQL | localhost:5432 | | Redis | localhost:6379 | | Kafka | localhost:9092 | | Redpanda Console | localhost:8088 | | RedisInsight | localhost:5540 | ## Testing ```bash pnpm test pnpm test:e2e ``` `pnpm test` runs the unit test suite. `pnpm test:e2e` runs the Playwright functional suite through an isolated local stack: it creates/resets `proofhound_e2e`, uses Redis DB 1, starts API, webhook, worker, web, and the fake LLM server, then stops the app processes after Playwright exits. The suite prefers API `http://localhost:4200`, webhook `http://localhost:4201`, web `http://localhost:3200`, and fake LLM port `5599`, but automatically selects nearby free ports when those defaults are occupied. To run a single e2e spec: ```bash pnpm test:e2e e2e/experiment.spec.ts --reporter=line ``` ## Configuration ProofHound reads the repo-root `.env` (used by server, webhook, worker, and the DB scripts; `apps/web` reads `NEXT_PUBLIC_*` from `apps/web/.env.local`). `cp .env.example .env` gives working local defaults — the common variables you may want to set: | Variable | What it's for | Default | | ------------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------- | | `MODEL_API_KEY_ENCRYPTION_KEY` | **Required** — encrypts stored model API keys at rest. Generate a real one with `openssl rand -base64 32`. | dev placeholder | | `DATABASE_URL` | PostgreSQL connection string. | Docker Compose Postgres | | `REDIS_URL` | Redis connection (rate limits + queues). | Docker Compose Redis | | `SERVER_PORT` | Server API port. | `4000` | | `WEB_PUBLIC_URL` | Web origin allowed for CORS. | `http://localhost:3000` | | `NEXT_PUBLIC_SERVER_URL` | Server URL the web app calls. | `http://localhost:4000` | | `WORKER_CONCURRENCY` | Per-process `llm` queue concurrency. | `64` | | `LOG_LEVEL` | Pino log level. | `debug` | More advanced / optional variables (deploy metadata, DB reset & seeding, tests, the `pnpm probe:model` script, connector demos) are documented inline in [`.env.example`](.env.example). ## Walkthrough Once ProofHound is running, the manual end-to-end flow from a fresh dataset to a production release is: 1. **Add a model** — register a provider/model with your endpoint, API key, pricing, and RPM / TPM / concurrency limits (start from a [quick preset](#models-and-providers)). 2. **Upload a dataset** — a CSV / TSV / JSONL / JSON / ZIP file, and map the field roles (input text/image, the expected output, metadata). 3. **Write a prompt** — create a prompt and its first version: the template, variables, output fields, and judgment rules. 4. **Run an experiment** — pick a prompt version × dataset × model and run the batch regression; read accuracy / precision / recall / F1, per-class metrics, failed samples, and the full run results. 5. **Inspect and iterate** — review the failed samples, edit the prompt into a new version, and re-run the experiment; repeat until the metrics hit your goal. (Each referenced version is frozen, so every comparison maps back to exact prompt content.) 6. **Release** — bind the winning version to an upstream connector and ship it: a queue connector goes through a canary (traffic split + dual-run) → 100% → production, with rollback and forced stop available; a webhook's first release goes straight to production. Run results are written for every call along the way, and you can layer human annotations on top without mutating them. ### Skip the manual loop with optimization Instead of doing step 5 by hand — reading failures, editing the prompt, and re-running round after round — create an **optimization** task: set a goal (e.g. a target accuracy, or recall on a specific class) and a round budget, and it automatically analyzes failed samples, generates new candidate versions, re-runs the regression each round, and keeps the best version. So with optimization you can **skip step 5** (it automates the loop), and via **Quick start** even **step 3** — the analysis model generates the first prompt version for you. You still do steps 1–2 (a model and a dataset; optimization also uses an analysis model) and step 6 (the release stays your call). ## How it works ProofHound is a TypeScript monolith split by module boundaries, with a Node.js worker for LLM calls. Three surfaces drive it — the Web UI, an HTTP API + MCP channel for agents and automation, and per-connector webhook ingress for online traffic — and they all share the same orchestration and storage.
