ArchiveBox
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- 29k
- Python
- MIT
- 14 h ago
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine
🎠An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika
r7 is a high-performance Java HTTP gateway core with composable routing, low-allocation design, and memory-mapped full audit logging with pluggable tailers.
Download media from X and keep a searchable local post library. Reversible reading controls, available as Chromium/Firefox extensions and a userscript.
Experiences converting Common Crawl's ARC files from the crawls 2008 - 2012 to the WARC format