Offline PII firewall for AI agents and LLM apps: fast local detection and redaction, Claude Code hook, LiteLLM guardrail. Zero network calls, one dependency.
Repo DataFog/datafog-python, read from GitHub 17 h ago. DataFog has not joined Take My Repo.
DataFog 4.12.0 carries structured PII detection through every engine, legacy
API and CLI anonymization path. Smart, spaCy and GLiNER modes retain the same
native card, IP and credential findings as the regex mode, then add NER spans.
TextService scans complete documents, and spaCy/GLiNER character offsets prevent
an earlier substring or repeated name from receiving the wrong redaction.
This release requires datafog-core~=1.5.0 in the base package and the rust
extra. Core 1.5 supplies complete IPv4-mapped IPv6 redaction, corrected email
field boundaries and Unicode mailboxes, and phone-plus-ZIP classification that
does not suppress the phone behind a false card finding. Native differences
are recorded in tests/contracts/rust-1.5.0.json. Capability contract 1 remains
unchanged; Python and Core have independent version numbers.
Lone surrogate characters no longer disable native scans. Dense overlap
suppression, Python fallback patterns and allowlist matching have bounded work;
typoed entity labels, invalid ranges and invalid locales raise explicit errors.
Anonymizing pipeline operations imply scanning. Async guardrail decorators
await their result, and non-string results raise TypeError.
LiteLLM scans nested tool/function arguments and prompt/input requests with a
shared token assignment for the request. Repeated values reuse tokens, and
existing markers do not collide. Streaming buffers the complete upstream
response before emitting its first redacted chunk, including text and tool
arguments split across deltas. This adds the full upstream completion latency
to first output. Streams exceeding eight million text characters or 100,000
chunks are rejected. Engine errors retain the configured open or closed
failure policy. Image and binary parts remain opaque and receive no OCR.
The Claude Code hook asks for review, or denies according to configuration,
when input exceeds its scan budget; it no longer silently permits an unscanned
tail. Unexpected hook failures retain the documented fail-open policy.
Hash APIs honor the selected algorithm while preserving existing plain-hash
formats. Engine APIs, the DataFog class and CLI retain SHA-256 with 12 hex
characters by default. Deprecated top-level datafog.process(method="hash")
retains MD5 with eight hex characters. The legacy Anonymizer model retains its
existing full plain digest. Explicit hash_key adds full-length HMAC with at
least 32 key bytes; omitting the algorithm in keyed mode selects SHA-256.
Unkeyed hashes remain guessable deterministic pseudonyms, not protection
against enumerating SSNs, phone numbers or other low-entropy values.
The minimum Pydantic version is 2.1. The nlp-advanced extra includes spaCy and
permits Torch 2.x versions compatible with Core's GLiNER2 extra. Base installs
show a CLI-extra installation hint instead of a missing-Typer traceback.
Unused duplicate implementations are removed, source distributions include
runnable tests and corpora, and CLI configuration output hides API keys.
Scheduled release jobs qualify candidates without publishing. Manual jobs
default to dry runs, reject stale or occupied versions, and record immutable
release refs before uploading artifacts.
The published Core 1.5.0 dependency passed installed-wheel qualification with
this Python release, including the native bridge and compatibility contracts.
The Python and Core versions are published independently.
Released: 2026-11-15
# Core package (lightweight) pip install datafog # With all features pip install datafog[all]
4.10.0 turns on the native engine. pip install datafog installs datafog-core
and detection uses it. Nothing is removed in this release and no call changes
shape; backend="python" restores the previous behaviour exactly.
Read the 4.10 migration guide before upgrading a
pipeline whose output is under test.
python -m pip install --upgrade "datafog==4.10.0"Wheels cover Linux (glibc and musl, x86_64/aarch64/s390x/ppc64le), macOS and
Windows (amd64/arm64). Elsewhere the source build needs Rust 1.88+. If Core
cannot be imported, detection falls back to the pure-Python engine and warns
once.
backend defaults to auto: native when datafog-core imports, PythonDATAFOG_BACKEND sets the default; backend="python" forces thePERSON and ADDRESS for prose now come from an embedded model compiled intodatafog-core โ no spaCy, no GLiNER, no model download, no interpreter.API_KEY, BEARER_TOKEN, JWT, PRIVATE_KEY,CREDENTIAL_URI, ACCOUNT_NUMBER, US_ROUTING_NUMBER, NPI, UUID and theUUIDdatafog.native exposes the native schema, capabilities(), the embeddeddatafog-core is pinned >=1.3.0,<2. Detection output is contract-versioned.Four reviewed differences from the Python engine are recorded in
tests/contracts/rust-1.3.0.json and asserted
by the suite: Core validates card checksums, requires non-alphanumeric SSN
boundaries, requires a North American area code starting 2-9 (which is what
keeps Unix timestamps out of PHONE), and rejects unadvertised locales instead
of ignoring them.
(555) 123-4567 and similar form data now match: Core keeps the area-code check
and accepts exchange codes starting with 1.
The frozen corpus runs twice in CI now: once against the Python engine, whose
expectations it encodes, and once against the native engine with its twelve
recorded divergences in
tests/contracts/native-corpus.json. The
native divergences are strictness differences (Core validates more) plus email
boundary handling that Core currently exposes through format: "env" /
"sql" settings โ flagged for a default decision in Core 1.4.
detect(), process(), OCR and Spark still work and still warn; all are removed
in 5.0. Their messages now name 4.10.0 as the last supported version and give
the exact pin. See the retirement schedule.
VERSIONING.md is new and defines what a version number promises here, with aDATAFOG_BACKEND=python trades that back for coverage.datafog[rust] remains as an explicit spelling of a dependency the baseAlpha Build: 2026-10-05
โ ๏ธ This is an alpha build for early testing. Expect rough edges.
# Core package (lightweight) pip install datafog # With all features pip install datafog[all]
4.9.0 bridges the existing Python API and DataFog Core. Python detection remains
the default. Existing imports, result classes, and legacy redaction strategies
continue to work. Rust detection and the native API preview are experimental,
explicit opt-ins.
# Base package: no native dependency is automatically installed. python -m pip install --upgrade "datafog==4.9.0" # Optional Rust backend and native API preview. python -m pip install --upgrade "datafog[rust]==4.9.0"
The Rust extra requires datafog-core>=0.4.0,<0.5 and capability contract 1.
Installing it does not select Rust automatically. The all extra does not
include Rust; request datafog[all,rust]==4.9.0 if both are needed. Lock your Core
version when detection output must be reproducible across installations.
import datafog result = datafog.redact("Contact jane@example.com", engine="regex", backend="rust") assert result.redacted_text == "Contact [EMAIL_1]"
Rust is supported only with engine="regex". Missing dependencies, unsupported
configurations, and native failures raise errors without silently falling back.
The CLI, DataFog, TextService, ML engines, and existing integrations retain
their current detection paths.
datafog.compat.v4 exposes existing scan/redact APIs and result classes withdatafog.v5 exposes native Core types and operations. Scanning returnslist[Finding]; native transformations return TransformResult, use Core'sPERSON is rejected for explicit Rust text selection.Rust is not advertised as universally equivalent to the Python detector. On the
frozen 111-case baseline there are 77 exact matches, two reviewed detector
differences, one validation-message difference, and 31 cases outside backend
scope. Core rejects invalid-checksum cards and alphanumeric-embedded SSNs in the
reviewed differing cases. Unsupported locale validation uses Core capabilities.
Explicit NPI selection has a compatibility limitation: when Core reports both
PHONE and NPI for the same span, legacy overlap handling keeps PHONE before
entity filtering. Consequently entity_types=["NPI"] can return no entities.
Default legacy redaction still protects that span as a phone. Native
datafog.v5.scan() retains NPI, and native transformation can select it. Choose
the native API when retaining NPI identity is required.
The adapter resolves overlaps before entity filtering. A later compatible Core
release can introduce a finding that wins an overlap and suppresses a previously
selected label. The dependency range promises API compatibility, not identical
detection or selection results. For reproducible behavior, pin both packages:
python -m pip install "datafog[rust]==4.9.0" "datafog-core==0.4.0"
Use native findings and native transformation entity selection when preserving
specific overlapping labels is required.
detect() and process() still work in 4.9 but now warn of removal in 5.0. This
revises the earlier promise to retain them throughout 5.x. Migrate to scan/redact
and review transformation differences, particularly older placeholder and hash
formats.
OCR, Donut, Tesseract, image services, Spark, and distributed helpers remain
functional in 4.9 with visible use-time warnings. Their removal is planned for
5.0. Users needing them can remain on the final 4.x release; this does not promise
indefinite maintenance or introduce successor packages. spaCy and GLiNER are not
removed by this release.
See the migration guide for API examples, compatibility
boundaries, parity evidence, and reproducible verification commands.
Alpha Build: 2026-09-29
โ ๏ธ This is an alpha build for early testing. Expect rough edges.
# Core package (lightweight) pip install datafog # With all features pip install datafog[all]