rag-cpp
A type-theoretic, production-grade retrieval (RAG) engine in modern C++23.
One library. No vector-DB dependency. Hybrid lexical + dense retrieval, HNSW
ANN with filtered search, cross-encoder reranking, pluggable embedders, a
composable stage pipeline, source loaders (files / HTML / PDF / code), a stable
versioned on-disk format, and a C ABI for Python/Rust/Go — on a foundation that
makes illegal states unrepresentable.
#include <rag/rag.hpp>
rag::Engine engine;
engine.with_embedder(rag::dense::AnyEmbedder{
rag::dense::OllamaEmbedder{{.model = "nomic-embed-text"}}}); // or Hash / OpenAI / llama.cpp
engine.add("intro.md", "rag-cpp fuses BM25 with dense vector search...");
engine.build();
for (auto& r : *engine.search("how does hybrid retrieval work?", 5))
std::printf("[%.3f] %s\n", r.score.get(), r.uri.c_str());
Why "type-theoretic"?
The library treats the type system as a proof assistant:
- Strong / phantom types.
DocId, ChunkId, TermId are nominally
distinct even though all wrap uint32 — you cannot pass one where another is
wanted. Score and Similarity are distinct float newtypes.
- Total functions. Every fallible op returns
Result<T> = std::expected<T, Error>; no exceptions for control flow. Errors are a closed Errc sum type.
- Structural interfaces via concepts.
Embedder, Retriever, Ranker,
Reranker, Tokenizer are concepts — plug in a backend by matching a shape,
zero vtables on the hot path. Runtime polymorphism (config-assembled
pipelines) rides separate type-erased AnyX adapters.
Features
Retrieval
- Hybrid BM25 (Okapi, smoothed idf) + dense cosine, fused with RRF
(weighted) or Relative Score Fusion.
- HNSW ANN (Malkov & Yashunin) with Matryoshka truncation + binary
quantization, and filtered-HNSW: a metadata predicate pushed into the
graph walk (pre-filter) so selective filters still return k results.
- Cross-encoder reranking as a first-class stage — an in-process ONNX
cross-encoder (no server), TEI
/rerank, Cohere/Jina /v1/rerank, or any
in-process scorer, with a blend knob to mix cross-encoder and retrieval
scores (measured: trusting an out-of-domain CE outright can hurt — see
BENCHMARKS).
- Query expansion (RM3-lite PRF) and parent-document stitching
(small-to-big) pipeline stages.
GraphRAG
- An explicit document graph: link edges (markdown
[..](..), bare
URLs, [[wikilinks]]) + similarity edges (dense centroid cosine, else
lexical Jaccard; k-NN sparsified).
- Communities via deterministic label propagation, summarized
extractively (free, no model) with an optional abstractive summarizer seam.
- Local search (hybrid seed → Personalized-PageRank graph expansion) and
Global search (rank community summaries) — the full GraphRAG recipe with
the expensive LLM ingredients made deterministic by default.
Cutting-edge retrieval
- SPLADE-style learned sparse — saturated impact weighting + query-time term
expansion over an inverted index (BM25 speed, dense-like recall).
- ColBERT late interaction — token-level MaxSim reranking between
bi-encoders and cross-encoders.
- RAPTOR — recursive cluster-summarize tree; retrieve the collapsed tree
across abstraction levels (Sarthi et al., ICLR 2024).
- HyDE — hypothetical-document embeddings + multi-query / RAG-Fusion
(Gao et al. 2022).
- Corrective RAG + Self-RAG — a retrieval evaluator grading confidence into
correct / ambiguous / incorrect actions, decompose-recompose knowledge strips,
external-source fallback, and an answer groundedness score (Yan et al.
2024; Asai et al. 2024).
Evaluation
A BEIR-format harness (corpus/queries jsonl + qrels tsv) computing
nDCG@k, Recall@k, Precision@k, MAP, MRR — so SOTA is a measured number, not
a claim. See examples/beir_eval.cpp.
Production
- Product Quantization — 4–64× embedding compression with ADC scoring.
- MMR diversity reranking — stop returning k paraphrases of one passage.
- Retrieval cascade — hybrid → ColBERT → cross-encoder with per-stage budgets.
- Semantic + proposition chunking and Contextual Retrieval (Anthropic
- situating context.
- Incremental delete — HNSW tombstones +
Corpus::remove_document with
stable ids and compact().
- Embedding + query LRU caches (thread-safe, identity-keyed).
ragcpp CLI — index / query / eval / info with no C++.
RALM assemblies (retrieval-augmented language modeling)
Generator-agnostic retrieval frontends for four landmark recipes (the LM call is
your seam):
- RAG / REPLUG — temperature-scaled softmax ensemble weights
p(z|x) over
retrieval scores + a distribution-mixing combinator.
- RETRO — chunked-neighbour retrieval with continuations (the chunk that
followed each neighbour).
- In-Context RALM — a retrieval stride schedule with a rerank hook.
- Grounded prompt assembly — numbered, source-attributed context.
Embedders (pluggable)
Ollama · OpenAI-compatible (+ Together, TEI presets) · llama.cpp
server · deterministic local Hash (no network). Decorators: RetryingEmbedder
(bounded retries), FallbackEmbedder (primary → secondary). All behind an
injectable HttpTransport seam — bring your own TLS/gRPC/mock.
In-process (no server): ONNX Runtime (OnnxEmbedder, with WordPiece
tokenization + mean/CLS/max pooling) and GGUF via llama.cpp (GgufEmbedder).
Opt in with -DRAGCPP_WITH_ONNX=ON / -DRAGCPP_WITH_LLAMA=ON; without the dep
they report unavailable and the core stays dependency-free.
Source loaders
Filesystem directory walker (include/exclude globs) · HTML → text · CSV row-wise
with columns as filterable metadata · Word, Excel and PowerPoint read
in-process with no dependency at all (rag-cpp owns a ZIP reader and a DEFLATE
decoder; Excel's shared-string table is resolved, table rows stay one line,
slides keep presentation order) · PDF and legacy .doc/.xls/.ppt/.rtf
via an external converter when one is installed · scanned PDFs detected and
reported by name rather than silently indexed as empty · register_extractor()
for formats internal to one company.
Code, including languages nobody has a parser for
A code-aware chunker that splits on function/class boundaries for C-like,
Python, Ruby, Go and Rust — and, for everything else, structure inference:
it reads a file once, learns the conventions it was written with, and chunks on
those. No grammar, no dependency, and it works the first time it sees a
language, which a grammar by construction cannot. Measured at 1.000
definition integrity vs 0.75–0.875 for fixed windows on five invented DSLs
that exist nowhere on the internet — while correctly declining on prose, so
a mixed corpus is safe. See
docs/formats.md and
BENCHMARKS.md.
Persistence
A stable, versioned, CRC-checked .ragdb container (documented in
FORMAT.md) — a public contract, not an internal cache. Reopening
never rebuilds.
C API / bindings
A flat opaque-handle C ABI (include/rag/c/rag.h) drives
the whole engine from any language. See examples/ragcpp.py
for ctypes bindings; Rust bindgen / Go cgo work the same way.
Build
Requires a C++23 compiler (GCC 13+, Clang 17+) and CMake 3.24+.
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build # C++ + C-API suites
./build/examples/ragcpp_full_pipeline # the maximal funnel
./build/examples/ragcpp_graphrag # GraphRAG + RALM assemblies
./build/examples/ragcpp_advanced # SPLADE, ColBERT, RAPTOR, HyDE, CRAG
./build/examples/ragcpp_beir_eval <dir> # BEIR nDCG/Recall/MAP/MRR
./build/bench/ragcpp_bench 5000 # ablation + latency
For the shared library (Python/Rust FFI): -DBUILD_SHARED_LIBS=ON. The only
dependency (nlohmann/json) is fetched via FetchContent.
CLI
The ragcpp binary builds, searches, and serves a corpus with no code:
ragcpp index ./docs corpus.ragdb # walk a directory, chunk, index, save
ragcpp info corpus.ragdb # document/chunk counts
ragcpp query corpus.ragdb "my question" -k 5 --mmr
ragcpp serve corpus.ragdb # a conformant RCP/1 server over stdio
ragcpp serve corpus.ragdb --http 8000 --write --graph # HTTP + writable index + GraphRAG
ragcpp eval ./nfcorpus --split=test # BEIR metrics
ragcpp serve opens a saved .ragdb and brings up an RCP/1
endpoint with zero C++ — L2-certified, driveable by any RCP client. --write
advertises a writable index (index/add+index/delete with upsert); --graph
advertises GraphRAG local/global search.
Use rag-cpp as a library
rag-cpp installs a relocatable CMake package, so consuming it from another
project is one find_package away — the dependency on nlohmann/json is resolved
for you:
find_package(ragcpp CONFIG REQUIRED)
target_link_libraries(app PRIVATE ragcpp::ragcpp)
cmake --install build --prefix /usr/local # or any prefix on CMAKE_PREFIX_PATH
Or vendor it directly with FetchContent / add_subdirectory — the same
ragcpp::ragcpp target name works in-tree and installed.
Serve over RCP (the Retrieval Context Protocol)
rag-cpp speaks RCP/1 — an open JSON-RPC
protocol for retrieval engines (what MCP is to tools, RCP is to grounding
context). Any RCP client can drive the engine's hybrid retrieval, embedding,
GraphRAG, and index over stdio or HTTP. The front-end is a thin, header-only
framework layer (rag/rcp/) over the same live Engine your app already builds:
#include <rag/rcp/rcp.hpp>
int main() {
rag::Engine engine; // your engine, your corpus
engine.with_embedder(/* … */);
engine.add("doc://1", "…"); engine.build();
rag::rcp::serve_stdio(engine); // now a conformant RCP/1 server
}
Need more surface? The fluent ServerBuilder advertises exactly the
capabilities you turn on. Because rag-cpp already has the machinery, you
attach the engine's own advanced components and the matching RCP methods light
up natively — no host glue:
rag::rcp::ServerBuilder(engine)
.named("docs", "1.0")
.with_index(/*writable=*/true) // index/add + index/delete (upsert)
.with_graph().with_memory() // GraphRAG + HippoRAG-style memory
.with_splade(rag::sparse::SpladeIndex::build(engine.corpus()).value())
.with_colbert(rag::late::hashed_token_embedder(64)) // embed/multi + colbert rerank
.with_reranker(my_cross_encoder) // rerank + retrieve.rerank
.with_generator(my_llm) // query/transform + retrieve.rewrite (HyDE)
.filter_on("lang", "keyword") // §8 metadata filtering
.serve_http(8000);
The engine is held by reference — the host keeps ingesting/persisting
through its own handle while the server reads the same live corpus. Every
capability is advertised honestly: embed only with an embedder, sparse
mode + embed/sparse only with a SPLADE index, rerank/transform only when a
reranker/generator is attached. The funnel invariant candidateK ≥ topN ≥ k is
wire-enforced, and every hit carries a citation for grounded generation.
The bundled ragcpp_rcp_server demo wires all twelve RCP methods to real
engine machinery — hybrid/dense/sparse retrieve, cross-encoder + ColBERT
rerank, HyDE/multi-query query/transform, embed/embed/sparse/embed/multi,
GraphRAG graph, index/add+index/delete, and memory/build+memory/recall
over the community graph.
Build and certify:
cmake --build build --target ragcpp_rcp_server # RAGCPP_WITH_RCP=ON (default)
./build/examples/ragcpp_rcp_server # stdio
./build/examples/ragcpp_rcp_server --http 8000 # HTTP
python3 ~/projects/rcp/conformance/check.py -- ./build/examples/ragcpp_rcp_server
# → CERTIFIED LEVEL: L2
Performance (5000 chunks, Apple M-series, NEON)
embed+build: ~3.7 s (one-time)
hybrid query: ~2.1 ms/query
bm25 only: ~0.18 ms/query
dense (HNSW): ~0.14 ms/query
Quality (BEIR, measured)
Speed is only half of it. The standard pipeline with all-MiniLM-L6-v2
attached in-process, evaluated through Pipeline::run — the same code path
Engine::search executes:
| SciFact nDCG@10 |
|
| BM25 (BEIR paper) |
0.665 |
| rag-cpp, no model at all |
0.6809 |
| ColBERTv2 |
0.693 |
| SPLADE++ |
0.710 |
| rag-cpp hybrid (MiniLM-L6) |
0.7347 |
| rag-cpp dense (bge-base) |
0.7563 |
Hybrid beats both of its own halves — BM25 alone scores 0.6800, MiniLM alone
0.6518 — on all three datasets tested, which is the fusion earning its keep
rather than a better retriever being picked. Swap MiniLM for bge-base and the
ceiling rises on every dataset (SciFact 0.7347→0.7563, ArguAna 0.3848→0.4455) —
and the best config flips to pure dense, because a strong embedder no longer
needs BM25 fused in. All measured, not asserted. Full tables, the
NFCorpus/ArguAna runs, the embedder-scaling comparison, and the reproduction
commands are in
BENCHMARKS.md.
Documentation
Full guides live in docs/. Start there if you're new.
Guides
- Getting Started — install, first index, first query (CLI + library).
- Retrieval — hybrid BM25 + dense, HNSW, fusion, filtering, quantization.
- Vector store — using rag-cpp as a plain vector DB (your ids, your vectors).
- The Pipeline — composable stages;
standard / quality / context factories; custom stages.
- Embedders — every backend, the retry/fallback decorators, the
HttpTransport seam.
- Advanced Retrieval — SPLADE, ColBERT, RAPTOR, HyDE, CRAG, GraphRAG.
- Configuration — every
CorpusConfig / HnswConfig tunable.
- The CLI — full
ragcpp command reference.
- Serving over RCP — turning an
Engine into a conformant RCP/1 server.
- Persistence — the
.ragdb container and the write-ahead log.
- The C API & bindings — the flat C ABI and driving it from Python/Rust/Go.
- GPU acceleration — the optional Metal batch-scoring backend.
Reference
BENCHMARKS.md — measured BEIR / ANN / throughput numbers, reproducible.
CHANGELOG.md — release notes.
ARCHITECTURE.md — layers, seams, invariants.
PLUGINS.md — the three extension axes: concepts, AnyX, and the
load-time plugin registry (register backends by name / from a shared library).
FORMAT.md — the .ragdb on-disk contract.
References
The design draws on the retrieval-augmentation literature:
- Lewis et al., Retrieval-Augmented Generation (NeurIPS 2020) — marginalize
the generator over retrieved documents weighted by
p(z|x).
- Borgeaud et al., RETRO (ICML 2022) — chunked cross-attention over
retrieved neighbours and their continuations.
- Ram et al., In-Context RALM (TACL 2023) — retrieve at a generation stride,
rerank, prepend; leave the LM untouched.
- Shi et al., REPLUG (NAACL 2024) — black-box LM ensembling over retrieved
documents.
- Edge et al., GraphRAG (Microsoft 2024) — entity/document graph →
communities → community summaries → graph-aware local + global search.
- Sarthi et al., RAPTOR (ICLR 2024) — recursive cluster-summarize tree,
collapsed-tree retrieval across abstraction levels.
- Khattab & Zaharia, ColBERT (SIGIR 2020) — late interaction (MaxSim) over
token embeddings.
- Formal et al., SPLADE (SIGIR 2021) — learned sparse retrieval with term
expansion.
- Gao et al., HyDE (2022) — precise zero-shot dense retrieval via
hypothetical documents.
- Yan et al., Corrective RAG (2024) and Asai et al., Self-RAG (ICLR
- — retrieval evaluation, correction, and reflective critique.
- Thakur et al., BEIR (NeurIPS 2021) — the zero-shot IR benchmark this
library's eval harness targets.
License
MIT.