title: Multi-Modal RAG
emoji: ๐
colorFrom: indigo
colorTo: purple
sdk: gradio
app_file: app.py
pinned: false
Multi-Modal RAG
Hybrid retrieval over PDFs, text and images, on free small models. Everything but the
LLM runs locally on CPU.
| Stage |
Model / Tech |
Size |
| Text embeddings |
BAAI/bge-small-en-v1.5 (ONNX) |
~130MB |
| Image embeddings |
clip-ViT-B-32 |
~600MB |
| Dense index |
turbovec IdMapIndex (TurboQuant, 4-bit) |
8x smaller than float32 |
| Sparse index |
SQLite FTS5, built-in BM25 |
โ |
| Fusion |
Reciprocal Rank Fusion (k=60) |
โ |
| Reranker |
ms-marco-MiniLM-L-6-v2 (ONNX, quint8) |
~90MB |
| OCR |
RapidOCR (PP-OCRv4 mobile, ONNX) |
~15MB |
| LLM |
Groq qwen/qwen3.6-27b |
API |
Pipeline
Ingest โ chunk (350 words, 60 overlap) โ embed โ SQLite + turbovec.
Query โ dense (turbovec, 3x over-fetch then exact rescore) โ BM25 (FTS5) โ RRF โ
cross-encoder rerank โ Groq answer with citations. Images are searched by CLIP in the
same query and shown in a gallery.
Scanned PDFs with no text layer are rendered at 200 DPI and OCR'd, so they stay
searchable as text rather than being reduced to pictures.
Storage
SQLite is the source of truth and holds the raw float32 vectors; the .tvim files are
a cache. Delete them and they rebuild on next start. Deleting a file removes its rows,
its vectors (turbovec remove is O(1)) and its extracted images.
turbovec quantization is lossy โ the dense stage fetches CANDIDATES * OVERSAMPLE
and rescores them exactly against the stored vectors, which measured 99.9% agreement
with brute-force cosine at 3x. Drop OVERSAMPLE to trade recall for latency.
Run
pip install -r requirements.txt
echo GROQ_API_KEY=your_key > .env
python app.py
python test_e2e.py # ingest, dedup, delete, restart, concurrency
python eval.py # retrieval metrics per stage
eval.py reports recall@1/@5 and MRR separately for the dense, BM25, fused and
reranked stages, so a regression points at the stage that caused it. Run it before and
after touching chunking, fusion or the index. Use your own data with
--dataset qa.jsonl ({"question":..., "source":..., "answer":..., "wrong":[...]} per line).
The built-in corpus includes near-miss documents โ same vocabulary, wrong numbers โ
because without them every stage scores a meaningless 1.000. Current measurements on
a low-end CPU:
| stage |
recall@1 |
recall@5 |
MRR |
| dense |
0.333 |
1.000 |
0.667 |
| bm25 |
0.333 |
1.000 |
0.667 |
| fused |
0.333 |
1.000 |
0.667 |
| reranked |
1.000 |
1.000 |
1.000 |
That gap is the reranker earning its place: first-stage retrieval puts a distractor
first two times out of three, and the cross-encoder fixes every one. Retrieval p50 is
~680ms, answer accuracy 6/6, distractor contamination 0/6.
Observability
Every query and ingest is traced with RAGObserve
into a local SQLite file. No server, no account.
ragobserve ui # dashboard at http://127.0.0.1:5601 โ run from this directory
All wiring lives in ragobserve_enabled.py; the rest of the codebase just calls obs.
Stages captured, using the manual SDK (this is a custom pipeline, so no framework adapter):
| Stage |
What lands in the dashboard |
| ingestion |
file name, size, content hash |
| chunking |
chunk texts, window size, overlap |
| embedding |
model, count, dimensions, duration (ingest and query) |
| retrieval |
logged twice โ once for turbovec, once for FTS5 BM25, each with its own scores |
| fusion |
the fused list plus both inputs, so you can see what RRF changed |
| reranking |
before/after ordering with rank-shift arrows |
| context_assembly |
the exact prompt the model received, chunks included |
| generation |
model, prompt, response, tokens โ cost auto-priced, finish_reason |
Because dense and sparse are logged as separate retrievers, the Hybrid Search
Explorer shows BM25 vs vector side by side, and Reranker Analytics shows the
rank shifts that produce the recall jump in the eval table above.
eval.py additionally logs ground truth, so the dashboard computes Precision/Recall@k,
MRR and nDCG from real traces โ the same numbers the harness prints, kept per-query.
Tracing is never load-bearing: ragobserve_enabled.py wraps every call, so a missing,
disabled, or broken backend degrades telemetry and leaves answers untouched (there is a
test for exactly this). Switch it off with RAGOBSERVE=0, point it at a shared server
with RAGOBSERVE_URI=http://host:5601.
Tuning
Everything is env-overridable: CHUNK_WORDS, CHUNK_OVERLAP, CANDIDATES,
OVERSAMPLE, RERANK_TOP_K, RERANK_MAX_LEN, IMAGE_TOP_K, IMAGE_MIN_SCORE,
BIT_WIDTH (2โ4), MAX_FILE_MB, LLM_MODEL, DATA_DIR.
RERANK_MAX_LEN is the main latency dial โ the cross-encoder dominates query time.
Measured on the eval corpus: 512 โ 1726ms, 256 โ 676ms, 128 โ 295ms, all at identical
recall. 256 is the default because the eval corpus puts every answer in the first
sentence, which flatters shorter limits; real documents bury facts deeper.
reasoning_effort="none" is deliberate: qwen3 otherwise spends the token budget
thinking and returns empty content on long contexts.
Deploy to HF Space
Create a Gradio Space, push these files, add GROQ_API_KEY as a Space secret. Set
DATA_DIR=/data with persistent storage enabled, otherwise the index resets on
restart. Pinned versions in requirements.txt are load-bearing โ see the comment there.
Known limits
- Single process. Fine for a Space; a multi-replica deploy needs a shared DB and index.
- BM25 and dense search are in-process, so ingest of a very large PDF blocks queries
behind the write lock.