A lightweight, high-performance model engine for local AI
β¨ Features
π§ Model Management - Download, cache, and organize AI models from Hugging Face
π Advanced Filtering - Search models with regex patterns and SQL-style queries
π» System Detection - Automatic GPU detection and resource reporting
π OpenAI-Compatible API - RESTful API with streaming support
Installation
Install with Cargo
Build from Source
# Clone the repository
git clone https://github.com/InftyAI/PUMA.git
cd PUMA
# Build the binary
make build
# The binary will be available at ./puma
./puma version
Quick Start
CLI Usage
# Download a model
puma pull qwen/qwen2.5-0.5b
# Run a model in an interactive chat
puma run qwen/qwen2.5-0.5b
# List all models
puma ls
# Inspect model details
puma inspect qwen/qwen2.5-0.5b
# Check system info
puma info
# Remove a model
puma rm qwen/qwen2.5-0.5b
API Server
# Start the inference server with a model
puma serve qwen/qwen2.5-0.5b
# Server will start on http://0.0.0.0:8000
# API endpoints:
# POST /v1/chat/completions
# POST /v1/completions
# GET /v1/models
# GET /v1/models/:model
# GET /health
Test the API:
# Health check
curl http://localhost:8000/health
# Chat completion
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen2.5-0.5b",
"messages": [{"role": "user", "content": "Hello!"}]
}'
# Or use the test script
./hack/scripts/test_api.sh
Commands
| Command |
Status |
Description |
pull <model> |
β
|
Download model from provider |
ls |
β
|
List models (supports regex, label filters) |
inspect <model> |
β
|
Show detailed model information |
rm <model> |
β
|
Remove model and cache |
info |
β
|
Display system information |
version |
β
|
Show PUMA version |
run <model> |
β
|
Run a model in an interactive chat |
serve <model> |
β
|
Start OpenAI-compatible API server with a model |
ps |
π§ |
List running models |
stop |
π§ |
Stop running model |
Advanced Usage
Pattern Matching
# Substring match
puma ls qwen
# Prefix match
puma ls "^inftyai/"
# Alternation
puma ls "llama-(2|3)"
Label Filtering
# Single filter
puma ls -l author=inftyai
# Multiple filters (AND condition)
puma ls -l author=inftyai,license=mit
# Combine pattern + filter
puma ls llama -l author=meta
Available filters: author, task, license, provider, model_series
Engine Tuning
Both run and serve accept flags to tune the inference engine (KV-cache pool,
block size, batch size, default token budget):
puma run qwen/qwen2.5-0.5b --max-batch-size 64 --default-max-tokens 256
See docs/configuration.md for the full list of flags and defaults.
API Server
PUMA provides an OpenAI-compatible API server for model inference.
Starting the Server
# Start server with a model (default: 0.0.0.0:8000)
puma serve qwen/qwen2.5-0.5b
# Custom host and port
puma serve qwen/qwen2.5-0.5b --host 127.0.0.1 --port 3000
# Model must be pulled first
puma pull qwen/qwen2.5-0.5b
Engine parameters (KV-cache, batch size, etc.) can be tuned with additional flags β see docs/configuration.md.
API Endpoints
Chat Completions (Recommended)
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen2.5-0.5b",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello!"}
],
"max_tokens": 100,
"temperature": 0.7
}'
Streaming (Server-Sent Events)
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen/qwen2.5-0.5b",
"messages": [{"role": "user", "content": "Tell me a story"}],
"stream": true
}'
List Models
# Returns the currently loaded model
curl http://localhost:8000/v1/models
Health Check
curl http://localhost:8000/health
# Returns: {"status":"ok"}
OpenAI Python Client
PUMA is compatible with the OpenAI Python SDK:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="dummy" # Not required
)
response = client.chat.completions.create(
model="qwen/qwen2.5-0.5b",
messages=[
{"role": "user", "content": "Hello!"}
]
)
print(response.choices[0].message.content)
Inspect Output
$ puma inspect qwen/qwen2.5-0.5b
name: qwen/qwen2.5-0.5b
kind: Model
spec:
author: qwen
model_series: qwen2
task: text-generation
license: APACHE-2.0
context_window: 32.77K
safetensors:
total: 494.03M
parameters:
bf16: 494.03M
provider: huggingface
cache:
revision: 060db6499f32faf8b98477b0a26969ef7d8b9987
size: 988.10 MB
path: ~/.puma/cache/huggingface/models--qwen--qwen2.5-0.5b
status:
created: 2 months ago
updated: 2 months ago
Model Management
- Database:
~/.puma/models.db (SQLite)
- Cache:
~/.puma/cache/ (model files)
Models are stored with lowercase names for case-insensitive matching.
Development
# Build
make build
# Run all tests
make test
# Test API manually
./hack/scripts/test_api.sh
Project Structure
puma/
βββ src/
β βββ api/ # OpenAI-compatible API
β βββ backend/ # Inference backends (Mock, MLX)
β βββ cli/ # Command implementations
β βββ downloader/ # HuggingFace download logic
β βββ registry/ # Model registry & metadata
β βββ storage/ # SQLite storage backend
β βββ system/ # System info detection
β βββ utils/ # Formatting & helpers
βββ tests/ # Integration tests
βββ hack/ # Development scripts
βββ Cargo.toml # Rust dependencies
βββ Makefile # Build commands
License
Apache-2.0
Star History