Summarize and query from a lot of heterogeneous documents. Any LLM provider, any filetype, advanced RAG, advanced summaries, scriptable, etc
Repo thiswillbeyourgithub/wdoc, read from GitHub 15 min ago. thiswillbeyourgithub has not joined Take My Repo.
This patch release adds custom LiteLLM model metadata (MiniMax recipes), plus documentation updates and a version bump.
#23] Add custom LiteLLM model metadata for models LiteLLM does not know yet
wdoc/utils/customs/add_extra_litellm_models_metadata.py registering MiniMax model recipes (MiniMax-M3, MiniMax-M2.7, ...) with their context sizes, pricing, modalities and reasoning capabilitiesapi.minimax.io) and CN (api.minimaxi.com) OpenAI/Anthropic endpointswdoc/wdoc.py at import time, covered by tests/test_litellm_models.py88796988] Bump version 5.2.0 โ 5.2.1fdd9d291] Updated Sphinx build command in README034f0c9c] Regenerated Sphinx docs
wdoc.utils.customswdoc.utils.loadersbumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py
README.md
docs/source/wdoc.utils.customs.rst
docs/source/wdoc.utils.loaders.rst
[4f53f45] by @thiswillbeyourgithub, 20 minutes ago:
Merge branch 'main' into dev
[5a6fd0a] by @thiswillbeyourgithub, 37 minutes ago:
Merge pull request #23 from octo-patch/octo/20260801-model-add-recvqs4oL2zDzm
Add current MiniMax model recipes
[87940fe] by @octo-patch, 13 days ago:
Address LiteLLM metadata review feedback
tests/test_litellm_models.py
wdoc/utils/customs/add_extra_litellm_models_metadata.py
wdoc/wdoc.py
tests/test_litellm_models.py
wdoc/utils/customs/litellm_models.py
wdoc/wdoc.py
PLACEHOLDER
bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py
.gitignore
Adds a regression test asserting the file handed to transcribe_audio_whisper
is .mp3 (never .ogg). It mocks torchaudio/ffmpeg/file_hasher and the
transcriber, and uses a unique file_hash to bypass the joblib doc-loaders
cache so it stays reproducible.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
tests/test_wdoc.py
wdoc/utils/loaders/local_audio.py
Also, the original litellm error was only logged as a warning and then lost
when the fallback failed. Wrap the fallback request so its failure re-raises
with BOTH the endpoint error and the original litellm error, so the true root
cause (often an auth/proxy misconfiguration) survives.
Adds two regression tests (call the undecorated function to bypass the joblib
doc-loaders cache): one asserts the fallback uploads the full file, one
asserts the raised error mentions both the endpoint and litellm errors.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
tests/test_wdoc.py
wdoc/utils/loaders/shared_audio.py
Now the message stays concise and the traceback is shown exactly once:
attached via exc_info on the warn path, or printed by the re-raise on the
crash paths. Also drop the now-unused traceback import.
Adds a regression test asserting the logged message is concise (no embedded
"Full traceback:" dump) and that crash mode does not double-attach the
traceback to the log record.
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com
tests/test_wdoc.py
wdoc/utils/loaders/init.py
PLACEHOLDER
bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
wdoc-skill/REFERENCE.md
wdoc/docs/help.md
wdoc/utils/loaders/shared_audio.py
bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py
setup.py
setup.py
wdoc/utils/loaders/online_media.py
PLACEHOLDER
bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py
setup.py
Polished wording, added the skill frontmatter, and removed em-dashes from
the reference content. Updated every reference to the old root SKILL.md
(bump_default_models.sh, README.md, CLAUDE.md, ARCHITECTURE.md) and the
Claude Code install instructions to the new three-file layout.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
ARCHITECTURE.md
CLAUDE.md
README.md
bump_default_models.sh
wdoc-skill/EXAMPLES.md
wdoc-skill/REFERENCE.md
wdoc-skill/SKILL.md
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
CLAUDE.md
SKILL.md
[a5473f4] by @thiswillbeyourgithub, 9 days ago:
fix(summarize): dump LLM output on empty content, fix recursion dedup
Follow-up to the previous empty-summary fix, addressing review feedback.
Empty message content: instead of retrying (which just reproduces the
same blank when the cause is structural), raise a ValueError that dumps
every attribute of the LLM generations and llm_output. For some
reasoning models litellm leaves the message content empty and puts the
answer in additional_kwargs['reasoning_content'], so the dump reveals
when we are simply reading the wrong field
(generations[0].text == message.content).
Recursion dedup bug: if summary_text not in recursive_summaries tested
membership against the dict's integer level keys, so it was always True
and forced recursion to stop after the first pass. Now compares against
recursive_summaries.values(), which also makes the "identical summary"
warning accurate.
doc_reading_length: default to 0.0 (float) instead of 0 (int) when a
doc has no doc_reading_time, otherwise wdocSummary construction crashes
under beartype (WDOC_TYPECHECKING=crash).
Tests: the empty-content test now asserts the generation dump is included;
adds a recursion test proving summarization continues past the first pass
(fails as 2 != 3 with the old predicate).
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
tests/test_wdoc.py
wdoc/utils/tasks/summarize.py
_summarize previouslyassert 'monkey' in '' inNow each chunk generation retries once while bypassing the cache on an
empty completion, and raises an explicit ValueError if it is still empty.
Token usage from every attempt is accumulated so the cost accounting
stays correct.
Adds a basic regression test that drives _summarize with a fake LLM
returning empty content and asserts the explicit error is raised.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
tests/test_wdoc.py
wdoc/utils/tasks/summarize.py
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
CLAUDE.md
SKILL.md
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
SKILL.md
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
SKILL.md
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
SKILL.md
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
SKILL.md
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
CLAUDE.md
_check_provider_key "mistral/" MISTRAL_API_KEY guard inMake the embed model overridable via WDOC_TEST_MISTRAL_EMBED_MODEL
(default mistral/mistral-embed, full id incl. provider) and add it to
_ALL_TEST_MODELS so the guard now matches and crashes early.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
tests/run_all_tests.sh
tests/test_wdoc.py
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
tests/test_karakeep.py
[14cb323] by @thiswillbeyourgithub, 10 days ago:
Merge branch 'karakeep_source' into dev
[3ce4a93] by @thiswillbeyourgithub, 11 days ago:
test(karakeep): make the api test self-contained via a bookmark lifecycle
Instead of depending on a pre-existing library and a KARAKEEP_TEST_SELECTOR env
var, the api test now creates its own temporary text bookmark (deterministic,
unlike a link whose html is crawled asynchronously), loads it through wdoc's
loader, asserts the round-trip, then deletes it (mirrors karakeep_python_api's
own create/delete lifecycle tests). It also asserts the live bookmark carries
the structural keys the FakeKarakeep fixtures encode, so the fast basic tests
cannot silently drift from the real schema. Drop the now-unneeded
KARAKEEP_TEST_SELECTOR guard from run_all_tests.sh.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
tests/run_all_tests.sh
tests/test_karakeep.py
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
ARCHITECTURE.md
README.md
wdoc/docs/examples.md
wdoc/docs/help.md
tests/run_all_tests.sh
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
tests/test_karakeep.py
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
wdoc/utils/load_recursive.py
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
wdoc/utils/loaders/karakeep.py
setup.py
wdoc/utils/misc.py
[d03299d] by @thiswillbeyourgithub, 11 days ago:
Merge branch 'zotero_source' into dev
[3df03dc] by @thiswillbeyourgithub, 12 days ago:
test(zotero): drive the api test from ZOTERO_COLLECTION_NAME
Replace the bespoke ZOTERO_TEST_SELECTOR env var with ZOTERO_COLLECTION_NAME,
which sits alongside the standard pyzotero ZOTERO_* credentials, so the api
test setup is consistent with the other env-driven tests. The test now
parametrizes over both attachment-text routes (fulltext and wdoc) against the
named collection, and a new basic test asserts the fan-out marks every
sub-document loading_failure='warn'. run_all_tests.sh guards on
ZOTERO_COLLECTION_NAME accordingly.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
tests/run_all_tests.sh
tests/test_zotero.py
Emit each zotero sub-document with loading_failure='warn' so a failure is
logged and skipped while the rest load; the batch still raises only if every
sub-document fails. Found while parsing a real 17-item collection where one
abstract-less book entry crashed the whole run.
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
wdoc/utils/load_recursive.py
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com
ARCHITECTURE.md
README.md
This release focuses on modular installation extras, CLI robustness improvements, and a sweep of bug fixes across summarization, logging, and setup.
--foo-bar โ --foo_bar) automatically ([e9bfb80])sys.argv mutation via ArgvState ([4fe38f2]); accept --yt_* as shorthand for --youtube_doc_* ([f800805])-orig), falling back to en/en-US ([0753b00])[02:17:33]) ([c0779a1])*DEEP BREATH*-style LLM intro artifacts from all top-level bullets, not just the first ([dd09942], [c837143])--debug/--verbose by exact argv token, not substring, preventing false positives from argument values ([e183729])WDOC_WHISPER_API_KEY handling when OPENAI_API_KEY is unset ([23b6b1f])openparse-download behind an import openparse probe ([f4445b4]); scope yt-dlp pre-release upgrade to [youtube] users only ([9242566])install_requires into modular extras [youtube], [audio], [anki], [office], [logseq], [full] ([7bb4744])
sys.argv mutations in ArgvState helper class ([1098157], [a22f56d])setup_cli_logging(), called only from __main__.py ([203ab6f])ArgvState helpers with unit tests ([6bc897a], [a22f56d])test_wdoc.py to run_all_tests.sh for faster fail ([dbf4410])test_parse_docx on HTTP 429 instead of failing ([a02d684])run_all_tests.sh ([b6a0dd8])uvx wdoc[full] usage throughout README and examples.md ([0f72eaf], [7653e9a])[anki] extra in Anki parse example ([0f72eaf])uvx ([d88c461])bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
wdoc/utils/tasks/summarize.py
README.md
tests/run_all_tests.sh
tests/test_parsing.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
wdoc/utils/prompts.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
wdoc/utils/prompts.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
wdoc/utils/tasks/summarize.py
wdoc/utils/tasks/summarize.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
setup.py
import openparse probeCo-Authored-By: Claude Opus 4.7 noreply@anthropic.com
setup.py
--user, which is wrongpip install -U --pre yt-dlp if it'sCo-Authored-By: Claude Opus 4.7 noreply@anthropic.com
setup.py
[audio] extra with a python_version>='3.13'[audio], so the conditional belongs there.python setup.py installpip install wdoc[audio], uv, pipx, etc.) which never ranCo-Authored-By: Claude Opus 4.7 noreply@anthropic.com
setup.py
magic.from_buffer on stdin bytes. That code pathmagic.from_file in batch_file_loader.py and pdf.py) work fineCo-Authored-By: Claude Opus 4.7 noreply@anthropic.com
setup.py
Wrap the handler installation in a setup_cli_logging() function that
is called explicitly from wdoc/main.py. Library users get whatever
loguru handlers the host already configured (since loguru is a
singleton, wdoc's records will flow through them automatically); CLI
users get the customized colorized stdout/stderr plus the rotated
file log.
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
wdoc/st
dcdecc86, 6cbf025c, e5d7840b, 9d7802aa)
[p.N] citations; Python fallback adds them to uncited top-level bullets[p.N, filename.pdf] format with shortest disambiguating pathcitation_url_template parameter turns page citations into clickable markdown links (e.g. {source}#page={page})8f72df69): WDOC_ID citations now render as [N](#document-N) with HTML anchors for in-page navigation14ac41e4, be3dced3)
WDOC_DEFAULT_MODEL and WDOC_DEFAULT_QUERY_EVAL_MODEL now default to openrouter/deepseek/deepseek-v4-pro and openrouter/deepseek/deepseek-v4-flashb555a904: Coerce int to float for CLI kwargs type checking (fixes Gradio UI sending integer values for float parameters)4015a3a9: Better removal of "deep breath" mentions in summarization prompts61c8238c, 0ea3d13d, b34e2e3f: Crash early at import time when required API keys (OPENROUTER_API_KEY, OPENAI_API_KEY, MISTRAL_API_KEY, WDOC_WHISPER_API_KEY) are missing1d07163c: Warn and skip instead of crash when whisper test hits a 502 error8380d8ad: Skip ollama embedding test when the ollama port is unreachablee672e092: Better test cleanup in run_all_tests.sh72819fab: bump_default_models.sh helper script โ dry-run by default, --apply to write; syncs model names across docs, README, SKILL, ARCHITECTURE, and docker/env.example9d7802aa, ec09cf1e, 71560800, c400738f, 96f9ac23: CLAUDE.md and ARCHITECTURE.md updated with new settings documentation requirements, sphinx-apidoc command, bump_default_models.sh usage, and citation feature docs057bb75f: Added DeepWiki badge to README3f59670e: SVG updated to remove outdated default model reference"}bumpver.toml
docs/source/conf.py
setup.py
wdoc/wdoc.py
tests/test_wdoc.py
tests/test_wdoc.py
tests/test_wdoc.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
tests/test_wdoc.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
tests/test_wdoc.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
ARCHITECTURE.md
CLAUDE.md
Routes the defaults through OpenRouter rather than calling the deepseek
provider directly. Applied via bump_default_models.sh --apply.
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
README.md
SKILL.md
docker/env.example
wdoc/docs/help.md
wdoc/utils/env.py
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
bump_default_models.sh
wdoc/docs/svg/summary.svg
Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com
ARCHITECTURE.md
README.md
SKILL.md
docker/env.example
wdoc/docs/help.md
wdoc/utils/env.py
tests/run_all_tests.sh
README.md
CLAUDE.md
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
CLAUDE.md
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
wdoc/wdoc.py
[1b08a1e] by @thiswillbeyourgithub, 4 weeks ago:
Merge branch 'enh_citations' into dev
[96f9ac2] by @thiswillbeyourgithub, 4 weeks ago:
added a CLAUDE.md file
Signed-off-by: thiswillbeyourgithub 26625900+thiswillbeyourgithub@users.noreply.github.com
CLAUDE.md
wdoc/utils/tasks/summarize.py
Developed with Claude Code.
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
README.md
tests/test_wdoc.py
wdoc/docs/examples.md
wdoc/docs/help.md
wdoc/utils/tasks/summarize.py
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
wdoc/utils/tasks/summarize.py
wdoc/wdoc.py
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
wdoc/utils/prompts.py
wdoc/utils/tasks/summarize.py
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
wdoc/utils/tasks/summarize.py
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com
wdoc/utils/tasks/query.py
wdoc/wdoc.py