Promptfoo: LLM evals & red teaming
馃摝 About this Python package
This is a lightweight wrapper that installs promptfoo via pip. It requires Node.js 22.22.0 or newer and executes npx promptfoo@latest under the hood.
馃挕 If you have Node.js installed, we recommend using npx promptfoo@latest directly for better performance:
npx promptfoo@latest init
npx promptfoo@latest eval
See the main project for the official npm package.
馃悕 Use this pip wrapper when you:
- Need to install via
pip for Python-only CI/CD environments
- Want to manage promptfoo with poetry/pipenv/pip alongside Python dependencies
- Work in environments where pip packages are easier to approve than npm
promptfoo is a developer-friendly local tool for testing LLM applications. Stop the trial-and-error approach - start shipping secure, reliable AI apps.
Website 路
Getting Started 路
Red Teaming 路
Documentation 路
Discord
Installation
Requirements
- Python 3.10+ (for this wrapper)
- Node.js 22.22.0 or newer (required to run promptfoo; Node.js 24 LTS is recommended)
Install from PyPI
Alternative: Use npx (Recommended)
If you have Node.js installed, you can skip the wrapper and use npx directly:
npx promptfoo@latest init
npx promptfoo@latest eval
This is faster and gives you direct access to the latest version.
Quick Start
# Install
pip install promptfoo
# Initialize project
promptfoo init
# Run your first evaluation
promptfoo eval
See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.
What can you do with Promptfoo?
- Test your prompts and models with automated evaluations
- Secure your LLM apps with red teaming and vulnerability scanning
- Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
- Automate checks in CI/CD
- Review pull requests for LLM-related security and compliance issues with code scanning
- Share results with your team
Here's what it looks like in action:

It works on the command line too:

It also can generate security vulnerability reports:

Why Promptfoo?
- 馃殌 Developer-first: Fast, with features like live reload and caching
- 馃敀 Private: LLM evals run 100% locally - your prompts never leave your machine
- 馃敡 Flexible: Works with any LLM API or programming language
- 馃挭 Battle-tested: Powers LLM apps serving 10M+ users in production
- 馃搳 Data-driven: Make decisions based on metrics, not gut feel
- 馃 Open source: MIT licensed, with an active community
How This Wrapper Works
This Python package is a thin wrapper that:
- Checks that a supported Node.js version is installed
- Executes
npx promptfoo@latest <your-args> (or uses globally installed promptfoo if available)
- Passes through all arguments and environment variables
- Returns the same exit code
The actual promptfoo logic runs via the official TypeScript package from npm. All features and commands work identically.
Python-Specific Usage
With pip
pip install promptfoo
promptfoo eval
With poetry
poetry add --group dev promptfoo
poetry run promptfoo eval
With requirements.txt
echo "promptfoo" >> requirements.txt
pip install -r requirements.txt
promptfoo eval
In CI/CD (GitHub Actions example)
- name: Setup Node.js
uses: actions/setup-node@v7
with:
node-version: "24"
- name: Install promptfoo
run: pip install promptfoo
- name: Run red team tests
run: promptfoo redteam run
Learn More
Troubleshooting
"ERROR: promptfoo requires Node.js"
The wrapper needs Node.js 22.22.0 or newer to run. Install a supported version, then verify it with node --version:
- macOS:
brew install node
- Ubuntu/Debian: Use NodeSource or nvm; some distribution repositories still offer older versions
- Windows: Download from nodejs.org
- macOS/Linux: With nvm, run
nvm install 24
Slow First Run
The first time you run promptfoo, npx downloads the latest version from npm (typically ~50MB). Subsequent runs use the cached version and are fast.
To speed this up, install promptfoo globally:
The Python wrapper will automatically use the global installation when available.
Version Pinning
By default, this wrapper uses npx promptfoo@latest. To pin a specific version:
export PROMPTFOO_VERSION=0.95.0
promptfoo --version
Or install a specific version globally:
npm install -g promptfoo@0.95.0
Telemetry
This wrapper collects anonymous usage telemetry to help improve the package, mirroring the telemetry in the main promptfoo project.
What is collected: A random anonymous user ID, wrapper version, Python version, and whether you're running in CI. If you've previously logged into promptfoo and your email is stored in ~/.promptfoo/promptfoo.yaml, that is also sent.
To opt out:
export PROMPTFOO_DISABLE_TELEMETRY=1
Contributing
For promptfoo features, docs, and bugs: contribute to promptfoo/promptfoo.
For pip wrapper issues (installation, Python shim, CI integration): open issues here. See CONTRIBUTING.md for development setup.
Join our Discord community for help and discussion.
License
MIT License - Same as promptfoo