0.107.1 • Published 3 months ago

promptfoo v0.107.1

Weekly downloads
-
License
MIT
Repository
github
Last release
3 months ago

Promptfoo: LLM evals & red teaming

npm npm GitHub Workflow Status MIT license Discord

promptfoo is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.

Quick Start

# Install and initialize project
npx promptfoo@latest init

# Run your first evaluation
npx promptfoo eval

See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.

What can you do with Promptfoo?

  • Test your prompts and models with automated evaluations
  • Secure your LLM apps with red teaming and vulnerability scanning
  • Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
  • Automate checks in CI/CD
  • Share results with your team

Here's what it looks like in action:

prompt evaluation matrix - web viewer

It works on the command line too:

prompt evaluation matrix - command line

It also can generate security vulnerability reports:

gen ai red team

Why promptfoo?

  • 🚀 Developer-first: Fast, with features like live reload and caching
  • 🔒 Private: Runs 100% locally - your prompts never leave your machine
  • 🔧 Flexible: Works with any LLM API or programming language
  • 💪 Battle-tested: Powers LLM apps serving 10M+ users in production
  • 📊 Data-driven: Make decisions based on metrics, not gut feel
  • 🤝 Open source: MIT licensed, with an active community

Learn More

Contributing

We welcome contributions! Check out our contributing guide to get started.

Join our Discord community for help and discussion.

@anthropic-ai/sdk@apidevtools/json-schema-ref-parser@googleapis/sheetsajvajv-formatsasyncbetter-sqlite3cache-managercache-manager-fs-hashchalkchokidarcli-progresscli-table3commandercompressioncorscsv-parsecsv-stringifydebouncededentdotenvdrizzle-ormexpressfast-deep-equalfast-xml-parserfastest-levenshteinglobhttp-zinquirerjs-rougejs-yamlmathjsnode-cachenunjucksopenaiopenerproxy-agentpython-shellreplicaterfdcsemversocket.iotsxundiciuuidwinstonwszodzod-validation-error
@everything-registry/sub-chunk-2482supaiprompt-iteration-assistantslimspecllm-scoutllm-scout-typesllmstat@twilio-alpha/assistants-evalgenkitx-promptfoo
0.103.8

5 months ago

0.103.9

5 months ago

0.102.1

6 months ago

0.102.2

6 months ago

0.102.3

6 months ago

0.102.4

6 months ago

0.102.0

6 months ago

0.101.2

6 months ago

0.101.0

6 months ago

0.101.1

6 months ago

0.99.0

7 months ago

0.99.1

6 months ago

0.104.3

4 months ago

0.104.4

4 months ago

0.104.0

4 months ago

0.104.1

4 months ago

0.104.2

4 months ago

0.103.4

5 months ago

0.103.5

5 months ago

0.103.6

5 months ago

0.103.7

5 months ago

0.103.0

6 months ago

0.103.1

6 months ago

0.103.2

6 months ago

0.103.3

5 months ago

0.97.0

7 months ago

0.106.1

3 months ago

0.106.2

3 months ago

0.106.3

3 months ago

0.106.0

3 months ago

0.98.0

7 months ago

0.105.0

4 months ago

0.105.1

4 months ago

0.95.0

7 months ago

0.100.3

6 months ago

0.100.4

6 months ago

0.100.5

6 months ago

0.100.6

6 months ago

0.100.0

6 months ago

0.100.2

6 months ago

0.103.18

4 months ago

0.103.17

5 months ago

0.103.16

5 months ago

0.103.15

5 months ago

0.103.14

5 months ago

0.103.13

5 months ago

0.103.12

5 months ago

0.103.11

5 months ago

0.103.10

5 months ago

0.96.0

7 months ago

0.96.1

7 months ago

0.96.2

7 months ago

0.103.19

4 months ago

0.107.0

3 months ago

0.107.1

3 months ago

0.94.6

8 months ago

0.94.5

8 months ago

0.94.1

8 months ago

0.94.2

8 months ago

0.94.3

8 months ago

0.94.4

8 months ago

0.94.0

8 months ago

0.81.3

10 months ago

0.81.4

10 months ago

0.81.5

10 months ago

0.81.0

10 months ago

0.81.1

10 months ago

0.81.2

10 months ago

0.93.2

8 months ago

0.93.3

8 months ago

0.93.0

8 months ago

0.93.1

8 months ago

0.82.0

9 months ago

0.87.0

9 months ago

0.91.0

9 months ago

0.91.1

9 months ago

0.91.2

8 months ago

0.91.3

8 months ago

0.88.0

9 months ago

0.92.3

8 months ago

0.92.0

8 months ago

0.92.1

8 months ago

0.92.2

8 months ago

0.89.3

9 months ago

0.89.4

9 months ago

0.89.0

9 months ago

0.89.1

9 months ago

0.89.2

9 months ago

0.85.0

9 months ago

0.85.1

9 months ago

0.85.2

9 months ago

0.86.0

9 months ago

0.86.1

9 months ago

0.90.0

9 months ago

0.90.1

9 months ago

0.90.2

9 months ago

0.90.3

9 months ago

0.87.1

9 months ago

0.83.1

9 months ago

0.83.2

9 months ago

0.83.0

9 months ago

0.84.0

9 months ago

0.84.1

9 months ago

0.62.1

1 year ago

0.62.0

1 year ago

0.78.3

10 months ago

0.78.0

10 months ago

0.78.2

10 months ago

0.78.1

10 months ago

0.74.0

11 months ago

0.70.0

11 months ago

0.70.1

11 months ago

0.67.0

12 months ago

0.63.0

1 year ago

0.63.2

1 year ago

0.63.1

1 year ago

0.75.2

10 months ago

0.79.0

10 months ago

0.75.1

10 months ago

0.75.0

10 months ago

0.71.1

11 months ago

0.71.0

11 months ago

0.68.3

12 months ago

0.68.2

12 months ago

0.68.1

12 months ago

0.68.0

12 months ago

0.64.0

12 months ago

0.60.0

1 year ago

0.76.1

10 months ago

0.72.2

11 months ago

0.72.1

11 months ago

0.76.0

10 months ago

0.72.0

11 months ago

0.69.2

11 months ago

0.69.1

11 months ago

0.65.2

12 months ago

0.65.1

12 months ago

0.69.0

11 months ago

0.65.0

12 months ago

0.80.0

10 months ago

0.80.1

10 months ago

0.80.2

10 months ago

0.61.0

1 year ago

0.80.3

10 months ago

0.73.9

11 months ago

0.73.8

11 months ago

0.73.5

11 months ago

0.77.0

10 months ago

0.73.4

11 months ago

0.73.7

11 months ago

0.73.6

11 months ago

0.73.1

11 months ago

0.73.0

11 months ago

0.73.3

11 months ago

0.73.2

11 months ago

0.66.0

12 months ago

0.59.0

1 year ago

0.59.1

1 year ago

0.58.1

1 year ago

0.58.0

1 year ago

0.57.0

1 year ago

0.57.1

1 year ago

0.56.0

1 year ago

0.55.0

1 year ago

0.54.1

1 year ago

0.54.0

1 year ago

0.53.0

1 year ago

0.52.0

1 year ago

0.51.0

1 year ago

0.50.1

1 year ago

0.50.0

1 year ago

0.49.3

1 year ago

0.49.2

1 year ago

0.49.1

1 year ago

0.49.0

1 year ago

0.48.0

1 year ago

0.47.0

1 year ago

0.46.0

1 year ago

0.45.2

1 year ago

0.45.1

1 year ago

0.45.0

1 year ago

0.44.0

1 year ago

0.43.1

1 year ago

0.43.0

1 year ago

0.42.0

1 year ago

0.41.0

1 year ago

0.40.0

1 year ago

0.39.1

1 year ago

0.39.0

1 year ago

0.38.0

1 year ago

0.37.1

1 year ago

0.37.0

1 year ago

0.36.0

1 year ago

0.35.1

1 year ago

0.35.0

1 year ago

0.34.1

1 year ago

0.34.0

1 year ago

0.33.3

1 year ago

0.33.2

1 year ago

0.33.1

1 year ago

0.33.0

1 year ago

0.32.0

2 years ago

0.31.2

2 years ago

0.30.2

2 years ago

0.30.1

2 years ago

0.30.0

2 years ago

0.29.0

2 years ago

0.27.1

2 years ago

0.27.0

2 years ago

0.25.2

2 years ago

0.25.1

2 years ago

0.25.0

2 years ago

0.31.1

2 years ago

0.31.0

2 years ago

0.28.1

2 years ago

0.26.3

2 years ago

0.28.0

2 years ago

0.26.2

2 years ago

0.24.4

2 years ago

0.26.1

2 years ago

0.26.0

2 years ago

0.26.5

2 years ago

0.28.2

2 years ago

0.23.1

2 years ago

0.23.0

2 years ago

0.24.3

2 years ago

0.24.2

2 years ago

0.24.1

2 years ago

0.24.0

2 years ago

0.20.1

2 years ago

0.20.0

2 years ago

0.19.0

2 years ago

0.17.2

2 years ago

0.19.1

2 years ago

0.17.3

2 years ago

0.19.2

2 years ago

0.17.4

2 years ago

0.19.3

2 years ago

0.17.5

2 years ago

0.17.6

2 years ago

0.17.7

2 years ago

0.17.8

2 years ago

0.17.9

2 years ago

0.11.0

2 years ago

0.13.0

2 years ago

0.13.1

2 years ago

0.15.0

2 years ago

0.17.0

2 years ago

0.17.1

2 years ago

0.21.4

2 years ago

0.21.3

2 years ago

0.21.2

2 years ago

0.21.1

2 years ago

0.9.0

2 years ago

0.5.0

2 years ago

0.7.0

2 years ago

0.5.1

2 years ago

0.21.0

2 years ago

0.18.1

2 years ago

0.18.2

2 years ago

0.18.3

2 years ago

0.18.4

2 years ago

0.12.0

2 years ago

0.14.0

2 years ago

0.14.1

2 years ago

0.16.0

2 years ago

0.14.2

2 years ago

0.18.0

2 years ago

0.10.0

2 years ago

0.22.0

2 years ago

0.8.1

2 years ago

0.8.0

2 years ago

0.8.3

2 years ago

0.8.2

2 years ago

0.6.0

2 years ago

0.4.0

2 years ago

0.3.0

2 years ago

0.2.2

2 years ago

0.2.1

2 years ago

0.2.0

2 years ago

0.1.0

2 years ago