0.107.1 • Published 11 months ago

promptfoo v0.107.1

Weekly downloads
-
License
MIT
Repository
github
Last release
11 months ago

Promptfoo: LLM evals & red teaming

npm npm GitHub Workflow Status MIT license Discord

promptfoo is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.

Quick Start

# Install and initialize project
npx promptfoo@latest init

# Run your first evaluation
npx promptfoo eval

See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.

What can you do with Promptfoo?

  • Test your prompts and models with automated evaluations
  • Secure your LLM apps with red teaming and vulnerability scanning
  • Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
  • Automate checks in CI/CD
  • Share results with your team

Here's what it looks like in action:

prompt evaluation matrix - web viewer

It works on the command line too:

prompt evaluation matrix - command line

It also can generate security vulnerability reports:

gen ai red team

Why promptfoo?

  • 🚀 Developer-first: Fast, with features like live reload and caching
  • 🔒 Private: Runs 100% locally - your prompts never leave your machine
  • 🔧 Flexible: Works with any LLM API or programming language
  • 💪 Battle-tested: Powers LLM apps serving 10M+ users in production
  • 📊 Data-driven: Make decisions based on metrics, not gut feel
  • 🤝 Open source: MIT licensed, with an active community

Learn More

Contributing

We welcome contributions! Check out our contributing guide to get started.

Join our Discord community for help and discussion.

@anthropic-ai/sdk@apidevtools/json-schema-ref-parser@googleapis/sheetsajvajv-formatsasyncbetter-sqlite3cache-managercache-manager-fs-hashchalkchokidarcli-progresscli-table3commandercompressioncorscsv-parsecsv-stringifydebouncededentdotenvdrizzle-ormexpressfast-deep-equalfast-xml-parserfastest-levenshteinglobhttp-zinquirerjs-rougejs-yamlmathjsnode-cachenunjucksopenaiopenerproxy-agentpython-shellreplicaterfdcsemversocket.iotsxundiciuuidwinstonwszodzod-validation-error
@everything-registry/sub-chunk-2482supaillm-scoutllm-scout-typesllmstat@twilio-alpha/assistants-evalgenkitx-promptfooprompt-iteration-assistantslimspec
0.103.8

1 year ago

0.103.9

1 year ago

0.102.1

1 year ago

0.102.2

1 year ago

0.102.3

1 year ago

0.102.4

1 year ago

0.102.0

1 year ago

0.101.2

1 year ago

0.101.0

1 year ago

0.101.1

1 year ago

0.99.0

1 year ago

0.99.1

1 year ago

0.104.3

1 year ago

0.104.4

1 year ago

0.104.0

1 year ago

0.104.1

1 year ago

0.104.2

1 year ago

0.103.4

1 year ago

0.103.5

1 year ago

0.103.6

1 year ago

0.103.7

1 year ago

0.103.0

1 year ago

0.103.1

1 year ago

0.103.2

1 year ago

0.103.3

1 year ago

0.97.0

1 year ago

0.106.1

12 months ago

0.106.2

12 months ago

0.106.3

12 months ago

0.106.0

12 months ago

0.98.0

1 year ago

0.105.0

12 months ago

0.105.1

12 months ago

0.95.0

1 year ago

0.100.3

1 year ago

0.100.4

1 year ago

0.100.5

1 year ago

0.100.6

1 year ago

0.100.0

1 year ago

0.100.2

1 year ago

0.103.18

1 year ago

0.103.17

1 year ago

0.103.16

1 year ago

0.103.15

1 year ago

0.103.14

1 year ago

0.103.13

1 year ago

0.103.12

1 year ago

0.103.11

1 year ago

0.103.10

1 year ago

0.96.0

1 year ago

0.96.1

1 year ago

0.96.2

1 year ago

0.103.19

1 year ago

0.107.0

11 months ago

0.107.1

11 months ago

0.94.6

1 year ago

0.94.5

1 year ago

0.94.1

1 year ago

0.94.2

1 year ago

0.94.3

1 year ago

0.94.4

1 year ago

0.94.0

1 year ago

0.81.3

1 year ago

0.81.4

1 year ago

0.81.5

1 year ago

0.81.0

1 year ago

0.81.1

1 year ago

0.81.2

1 year ago

0.93.2

1 year ago

0.93.3

1 year ago

0.93.0

1 year ago

0.93.1

1 year ago

0.82.0

1 year ago

0.87.0

1 year ago

0.91.0

1 year ago

0.91.1

1 year ago

0.91.2

1 year ago

0.91.3

1 year ago

0.88.0

1 year ago

0.92.3

1 year ago

0.92.0

1 year ago

0.92.1

1 year ago

0.92.2

1 year ago

0.89.3

1 year ago

0.89.4

1 year ago

0.89.0

1 year ago

0.89.1

1 year ago

0.89.2

1 year ago

0.85.0

1 year ago

0.85.1

1 year ago

0.85.2

1 year ago

0.86.0

1 year ago

0.86.1

1 year ago

0.90.0

1 year ago

0.90.1

1 year ago

0.90.2

1 year ago

0.90.3

1 year ago

0.87.1

1 year ago

0.83.1

1 year ago

0.83.2

1 year ago

0.83.0

1 year ago

0.84.0

1 year ago

0.84.1

1 year ago

0.62.1

2 years ago

0.62.0

2 years ago

0.78.3

2 years ago

0.78.0

2 years ago

0.78.2

2 years ago

0.78.1

2 years ago

0.74.0

2 years ago

0.70.0

2 years ago

0.70.1

2 years ago

0.67.0

2 years ago

0.63.0

2 years ago

0.63.2

2 years ago

0.63.1

2 years ago

0.75.2

2 years ago

0.79.0

2 years ago

0.75.1

2 years ago

0.75.0

2 years ago

0.71.1

2 years ago

0.71.0

2 years ago

0.68.3

2 years ago

0.68.2

2 years ago

0.68.1

2 years ago

0.68.0

2 years ago

0.64.0

2 years ago

0.60.0

2 years ago

0.76.1

2 years ago

0.72.2

2 years ago

0.72.1

2 years ago

0.76.0

2 years ago

0.72.0

2 years ago

0.69.2

2 years ago

0.69.1

2 years ago

0.65.2

2 years ago

0.65.1

2 years ago

0.69.0

2 years ago

0.65.0

2 years ago

0.80.0

2 years ago

0.80.1

2 years ago

0.80.2

1 year ago

0.61.0

2 years ago

0.80.3

1 year ago

0.73.9

2 years ago

0.73.8

2 years ago

0.73.5

2 years ago

0.77.0

2 years ago

0.73.4

2 years ago

0.73.7

2 years ago

0.73.6

2 years ago

0.73.1

2 years ago

0.73.0

2 years ago

0.73.3

2 years ago

0.73.2

2 years ago

0.66.0

2 years ago

0.59.0

2 years ago

0.59.1

2 years ago

0.58.1

2 years ago

0.58.0

2 years ago

0.57.0

2 years ago

0.57.1

2 years ago

0.56.0

2 years ago

0.55.0

2 years ago

0.54.1

2 years ago

0.54.0

2 years ago

0.53.0

2 years ago

0.52.0

2 years ago

0.51.0

2 years ago

0.50.1

2 years ago

0.50.0

2 years ago

0.49.3

2 years ago

0.49.2

2 years ago

0.49.1

2 years ago

0.49.0

2 years ago

0.48.0

2 years ago

0.47.0

2 years ago

0.46.0

2 years ago

0.45.2

2 years ago

0.45.1

2 years ago

0.45.0

2 years ago

0.44.0

2 years ago

0.43.1

2 years ago

0.43.0

2 years ago

0.42.0

2 years ago

0.41.0

2 years ago

0.40.0

2 years ago

0.39.1

2 years ago

0.39.0

2 years ago

0.38.0

2 years ago

0.37.1

2 years ago

0.37.0

2 years ago

0.36.0

2 years ago

0.35.1

2 years ago

0.35.0

2 years ago

0.34.1

2 years ago

0.34.0

2 years ago

0.33.3

2 years ago

0.33.2

2 years ago

0.33.1

2 years ago

0.33.0

2 years ago

0.32.0

2 years ago

0.31.2

2 years ago

0.30.2

2 years ago

0.30.1

2 years ago

0.30.0

2 years ago

0.29.0

2 years ago

0.27.1

2 years ago

0.27.0

2 years ago

0.25.2

2 years ago

0.25.1

2 years ago

0.25.0

2 years ago

0.31.1

2 years ago

0.31.0

2 years ago

0.28.1

2 years ago

0.26.3

2 years ago

0.28.0

2 years ago

0.26.2

2 years ago

0.24.4

2 years ago

0.26.1

2 years ago

0.26.0

2 years ago

0.26.5

2 years ago

0.28.2

2 years ago

0.23.1

2 years ago

0.23.0

2 years ago

0.24.3

2 years ago

0.24.2

2 years ago

0.24.1

2 years ago

0.24.0

2 years ago

0.20.1

3 years ago

0.20.0

3 years ago

0.19.0

3 years ago

0.17.2

3 years ago

0.19.1

3 years ago

0.17.3

3 years ago

0.19.2

3 years ago

0.17.4

3 years ago

0.19.3

3 years ago

0.17.5

3 years ago

0.17.6

3 years ago

0.17.7

3 years ago

0.17.8

3 years ago

0.17.9

3 years ago

0.11.0

3 years ago

0.13.0

3 years ago

0.13.1

3 years ago

0.15.0

3 years ago

0.17.0

3 years ago

0.17.1

3 years ago

0.21.4

2 years ago

0.21.3

2 years ago

0.21.2

2 years ago

0.21.1

2 years ago

0.9.0

3 years ago

0.5.0

3 years ago

0.7.0

3 years ago

0.5.1

3 years ago

0.21.0

2 years ago

0.18.1

3 years ago

0.18.2

3 years ago

0.18.3

3 years ago

0.18.4

3 years ago

0.12.0

3 years ago

0.14.0

3 years ago

0.14.1

3 years ago

0.16.0

3 years ago

0.14.2

3 years ago

0.18.0

3 years ago

0.10.0

3 years ago

0.22.0

2 years ago

0.8.1

3 years ago

0.8.0

3 years ago

0.8.3

3 years ago

0.8.2

3 years ago

0.6.0

3 years ago

0.4.0

3 years ago

0.3.0

3 years ago

0.2.2

3 years ago

0.2.1

3 years ago

0.2.0

3 years ago

0.1.0

3 years ago