FFAI
The AI media toolkit, remade with Rust. Speech recognition, text-to-speech,
OCR, object detection and tracking, and vision-language captioning in one
toolkit. It is built the way ffmpeg was built: libraries first, one binary on
top, every engine swappable by name. No Python runtime, no ONNX runtime, no
gated weights, and the same engines compile to WebAssembly and run in a browser
tab.
Part of Remade With Rust.
In The Wild with 827,581 Active Installs
RAG Converter runs FFAI's engines as WebAssembly in
the browser to make personal and business files AI-readable: emails, PDFs,
scans, images, audio and video, converted to RAG data in real time with
nothing uploaded and nothing installed. An API serves the same pipeline to
businesses.
DLDeploy.com delivers browser-based AI built from the
Remade With Rust portfolio, with FFAI as the AI layer.
MATA Network builds Disco Party, the sovereign
distributed cloud, on the Remade With Rust stack. FFAI is that stack's AI
layer, so the apps deployed onto Disco transcribe, speak, read and see
without sending data to a third-party AI service.
Build on it with our
AI skills for Rust.
ffai asr -i talk.wav -o talk.srt # speech to subtitles
ffai asr -i talk.wav -o talk.json --word-timestamps # per-word times
ffai asr -i meeting.wav --diarize --max-speakers 3 # who spoke when
ffai tts "Hello from FFai." -o hello.wav # piper voices, pure Rust
ffai ocr -i page.png # documents
ffai ocr -i filing.pdf --auto-orient # PDFs, scans, forms
ffai ocr --live --watch 5 -i captures/ -o screen.srt # live screen reading
ffai detect -i street.jpg --engine yolo26n # YOLO26 detection
ffai detect --track -i frames/ -o tracks.txt # multi-object tracking
ffai caption -i frame.png --prompt "what is happening here?"
ffai engines # every engine and its status, like `ffmpeg -codecs`
ffai models # model manifests, licenses, cache status
Speed
Each engine is measured against the implementation people already deploy, with
the same weights, the same inputs and the same decoding settings, on CPU.
| Engine |
Task |
Against |
Speed |
Memory |
Quality |
| Mercury |
ASR |
whisper.cpp |
1.07× faster on clean speech, 1.70× on noisy |
0.84–0.92× |
WER 7.27 % vs 7.58 % |
| Mercury |
TTS |
piper |
1.58× faster at 5 % less CPU, 10× faster load |
parity |
WER 5.49 % vs 5.27 % |
| Carmenta |
OCR recognition |
PaddleOCR mobile |
2.6× lower latency |
— |
CER 1.5 % vs 3.0 % |
| Carmenta |
Document OCR |
Unlimited-OCR (3B, GPU) |
~9× throughput, on CPU |
MB of weights vs 6.4 GB |
edit distance 0.108 vs 0.041 |
| Carmenta |
Live screen OCR |
Tesseract per frame |
p95 257 ms vs 343 ms |
flat over 30 min |
0 churn vs 24 |
| Diana |
Detection |
Ultralytics |
3.71× less CPU per frame, 1.5–2.4× throughput |
1.6–5.6× leaner |
mAP50 within 0.08 pp |
| Diana |
Tracking |
Ultralytics ByteTrack |
— |
no ReID model |
218 ID switches vs 800 |
| Argus |
Captioning |
transformers |
preprocessing 2.6× faster; whole caption 1.20× slower |
— |
caption byte-identical |
The latest releases do the same work with less of it. These are
deterministic counts (callgrind instructions on one core, or bytes written),
not timings, and the output stays byte-identical. Wall time on the new
releases has not been re-measured, so the speed column above comes from the
previous release of each engine.
| Release |
Workload |
Work counted |
ffai-diana 0.10 |
detection, yolo26n at 640 |
−34.6 % (1,075 M → 703 M) |
ffai-diana 0.10 |
tracker frame |
−24.3 % |
ffai-argus 0.10.1 |
prefill / decode token / whole caption |
−24.2 % / −15.3 % / −8.7 % |
ffai-argus 0.10.1 |
preprocessing / prompt assembly |
−85.1 % / −58.7 % |
ffai-mercury 1.2 |
ASR zero-fill memset |
−99.4 % (573 MB → 3.3 MB) |
ffai-mercury 1.2 |
TTS zero-fill memset |
−93.1 % (503 MB → 34 MB) |
The decode path is ours too: rusty_jpeg decodes 1080p in 7.05 ms, within
14 % of libjpeg-turbo; rusty_png is byte-identical to upstream png; and
rusty_h264 decodes x264's default profile at 22.7 ms per 1080p frame.
Every figure traces to a line in bench/ledger.jsonl,
produced by ffai bench (below).
Components
| Component |
Crate |
Task |
Namesake |
| Mercury |
ffai-mercury |
ASR + TTS |
Roman god of language and messages |
| Carmenta |
ffai-carmenta |
OCR |
Roman goddess who adapted the Greek alphabet into Latin letters |
| Diana |
ffai-diana |
Detection + tracking |
Roman goddess of the hunt |
| Argus |
ffai-argus |
VLM captioning |
Argus Panoptes, the all-seeing watchman |
Browser builds: ffai-wasm (Diana),
ffai-carmenta-wasm, ffai-mercury-wasm and ffai-argus-wasm.
Infrastructure: ffai-core (engine traits, registry and shared fast math, on
candle), ffai-media (ingest and egress via
remade_ffmpeg_rs),
ffai-models (weight manifests and cache), ffai-bench (the analyzer),
ffai-cli (the ffai binary), and ffai-demo, a three-tab demo: Listen
runs Mercury beside whisper.cpp on one microphone, Speak shows phonemes and
a same-seed determinism check, Read runs OCR on an image you drop in
(cargo run --release -p ffai-demo).
Mercury: speech recognition and synthesis
ASR runs OpenAI Whisper, tiny to medium, on candle with our own mel
front-end, tokenizer grammar, decode loop and AVX2 kernels. It passes all four
gates (correctness, quality, speed, footprint) against whisper.cpp on both
LibriSpeech holdouts, 134 clips each, tiny.en, greedy:
| Corpus |
Implementation |
WER % |
×realtime |
steady MiB |
| test-clean |
Mercury |
7.27 |
27.8 |
179 |
| test-clean |
whisper.cpp |
7.58 |
25.9 |
195 |
| test-other |
Mercury |
16.89 |
33.3 |
163 |
| test-other |
whisper.cpp |
16.82 |
19.6 |
194 |
Most of the speed comes from adaptive encoder context: Whisper pads every
window to 30 s, so each window is encoded at a context sized to the audio it
actually holds, with guards that fall back to the full context. Stage by stage
it wins every stage against whisper.cpp with flash attention on: encode about
2.0×, decode 1.1–1.2×, mel 1.4×.
- Accuracy tiers:
small.en reaches 3.05 % WER, ahead of whisper.cpp
small.en (3.38 %) at 4.2× against 3.7× realtime. Beam search is available
with FFAI_BEAM_SIZE=5.
- The WhisperX layer, without Python or gated weights: speech segmentation
on by default (2.2–4.2× faster on audio with trailing silence, identical
transcript);
--word-timestamps by CTC forced alignment (100 % containment);
--diarize with ECAPA-TDNN (DER 4.21 %); and speaker labels that persist
across calls for live streams (streaming DER 5.68 %).
TTS runs the full VITS stack on candle, reading
piper's own voice files straight
from ONNX, with a clean-room pure-Rust phonemizer. Nothing GPL ships.
- Oracle-exact against piper's runtime: waveform to 3e-5, durations
integer-exact.
- 1.58× faster wall-clock at 5 % less CPU, model load 0.64 s against
6.69 s, memory at parity.
- Deterministic: the same text and seed give one exact WAV per machine
(1.2.1 pins the GEMM cache detection that hybrid P-core/E-core CPUs used to
split). Piper samples noise inside its graph and cannot offer this.
- 1.2 cut 45 pieces of redundant work across ASR and TTS with byte-identical
output, including 99.4 % of ASR's and 93.1 % of TTS's buffer zero-filling.
- English (en-US) voices.
Carmenta: OCR for documents, forms, PDFs and live screens
Five engines from two detector lineages and three recognizers, each
oracle-matched against its reference. mobiledet-svtr is the document
default and craft-crnn the live default.
- Recognition beats PaddleOCR's mobile recognizer on 400 real-photo word
crops: 1.5 % vs 3.0 % CER, 93 % vs 88 % exact match, at 2.6× lower
latency.
- Documents, scored by
OmniDocBench's official
evaluator on all 1,651 pages: text edit distance 0.116, reading order
0.204, inside the published range (0.033–0.157). On CPU it runs at about
9× the throughput of the 3B record holder Unlimited-OCR, which scores
higher (0.041 vs 0.108).
- Live mode emits text only when it changes: zero churn across 156
unchanged frames where per-frame engines churned 24 times, 24/24 changes
caught, p95 257 ms, memory flat over a 30-minute soak.
- Forms and PDFs: auto-orientation (144/144 quarter turns undone),
checkboxes (100 % found and read on 960 boxes), label-to-value pairing
(220/221), and PDF pages read as a viewer shows them: redactions are never
read through.
Diana: YOLO26 detection and tracking
The real YOLO26 graph on candle, all five tiers, loaded from the official
Ultralytics .pt. No ONNX, no Python at inference. Weights are AGPL-3.0 and
are never in this repository.
Graded on a 450-image COCO holdout against Ultralytics at the same
configuration:
| Tier |
mAP50 Δ (pp) |
Diana MiB |
Ultralytics MiB |
leaner |
| n |
−0.01 |
71 |
403 |
5.6× |
| s |
−0.08 |
110 |
447 |
4.1× |
| m |
+0.06 |
178 |
544 |
3.1× |
| l |
+0.06 |
197 |
535 |
2.7× |
| x |
−0.07 |
346 |
678 |
2.0× |
At four tiers the detections are identical in count, class and order, within
0.3 px. On all seven MOT17 training sequences (5,316 frames) AP50 matches
Ultralytics to a mean 0.029 pp.
- Speed: 3.71× less CPU per frame (84.6 ms against 314.2). On a
dedicated machine with each engine pinned to its own cores, wall-clock is
1.30× faster; under 16 competing threads it is 1.92×; batch throughput
is 1.5–2.4× at every tier.
- 0.10: 34.6 % fewer instructions per detection, from removed copies,
fused passes, Winograd 3×3 convolutions and a vectorized softmax. All five
tiers still pass the full-graph oracle against the official model.
- Tracking: ByteTrack with no appearance model, so it adds no download and
no license. 218 ID switches against Ultralytics' 800 on MOT17, IDF1
35.93 % vs 36.92 %.
- In the browser:
ffai-wasm runs
the whole graph as a 1.82 MB module with an 18 ms model load. Every box and
class matches native. crates/ffai-wasm/demo.html is a working page.
- Lean to embed: 138 transitive crates natively, 95 on wasm32, and no
dependency on the other engines.
- Standalone page: Remade-With-Rust/diana.
Argus: vision-language captioning
SmolVLM-256M-Instruct on candle, with our own SigLIP encoder and SmolLM2 text
tower. From a raw image file, the caption matches the Python reference byte
for byte, including preprocessing, which implements PIL's fixed-point
resampler exactly.
- 0.10.1 executes 24.2 % fewer instructions per prefill, 15.3 % fewer per
decoded token, 8.7 % fewer per caption, 85.1 % fewer in preprocessing and
58.7 % fewer in prompt assembly. The caption, the decode tokens and the
preprocessed bytes are unchanged. Wall time has not been re-measured since
0.10.0.
- On 0.10.0, preprocessing is 2.6× faster than
transformers, and a whole
caption is 1.20× slower, with the gap in the vision tower.
- Video captions stream to
.srt, .vtt or .json one window at a time, so
memory is set by --window, not by clip length.
ffai caption -i street.jpg --prompt "What is written in this image?"
ffai caption -i clip.mp4 --fps 2 --window 8 --output captions.srt
The analyzer: ffai bench
One command compares any FFAI engine against world-standard implementations on
a pinned, hash-verified corpus:
ffai bench asr --corpus corpora/librispeech-test-clean-v2.toml
ffai bench asr --corpus corpora/librispeech-test-other-v1.toml --baseline-only
It runs our engine and every reference in corpora/references.toml over the
same clips, scores both with the same metric code, and appends an audit record
to bench/ledger.jsonl: corpus fingerprint, reference versions, environment,
and a four-gate verdict (correctness, quality, speed, footprint) in which a
skipped gate is never a pass. Methodology:
docs/benchmarking.md.
Architecture
One trait per task, many engines per trait, selected by name.
AsrEngine, TtsEngine, OcrEngine and VlmEngine are the AVCodec of
FFAI, and --engine whisper-candle is -c:v libx264. New models are new
engines, not rewrites.
- Pure Rust. GPU and accelerator backends sit behind feature flags.
- Library-first. The CLI contains no logic; every capability is a crate
you can embed.
- Candle is the tensor spine. One
Tensor and Device across all
engines.
- Weights are data, not code. Never vendored; fetched into a cache from
TOML manifests that surface each model's own license. Every model is
fetchable without an account.
- Streaming-first. Engines process chunks; whole-file is the degenerate
case.
- Oracle-gated. An engine is
stable only when validated against its
reference implementation on public corpora.
- Codecs come from home. Media routes through
remade_ffmpeg_rs;
images decode through
rusty_jpeg
and rusty_png.
- Pure-Rust allocator. The
ffai binary and the wasm crates use
rusty_alloc 2.4.0. It replaced
mimalloc with byte-identical output and speed parity, and the engines'
byte-identity oracles pass on every release.
Supply chain. Workspace builds patch
tokenizers onto
rusty_expressions,
our pure-Rust Oniguruma, so no C regex engine is compiled. Two C components
remain: crates taken from crates.io still pull onig_sys through candle until
the patch lands upstream, and aws-lc-sys sits behind the optional
Hugging Face downloader (ffai-models/fetch). The wasm32 builds compile no C.
Install
cargo install ffai-cli # the `ffai` binary
cargo add ffai-mercury # or embed any component as a library
Weights are fetched into a local cache on first use, or ahead of time with
ffai models --fetch <name>.
Build from source
cargo build --release
cargo test --workspace
target/release/ffai engines
Development history
The full campaign record behind every number, including refuted experiments
and corrections, stays in the repository:
Mercury,
Mercury TTS,
Carmenta,
Diana,
Argus, the instruction-count ledgers in
docs/plans/, and the root-cause write-ups in
docs/whys/. See ROADMAP.md for what is next.
License
MIT OR Apache-2.0 (code). Model weights carry their own licenses; check
ffai models.