ModFetch
A CLI toolkit for AI agents that fetches remote web pages and converts them into structured text evidence, bridging the web-reading gap for LLM workflows.
中文说明
Features
- Built for agent workflows that need direct web page content
- Security-first defaults for remote URL fetching (SSRF-oriented guardrails)
- Extracts readable text from HTML and returns machine-consumable JSON
- Single responsibility: fetch only (no web search, no image parsing)
- Designed to be called from Agent Skills (Claude Code, Codex, Cursor, etc.)
Install
npm install -g @liustack/modfetch
Or run with npx:
npx @liustack/modfetch [options]
Usage
# Print JSON result to stdout
modfetch -u https://example.com
# Save to file
modfetch -u https://example.com -o page.json
# Tune limits
modfetch -u https://example.com --timeout 30000 --max-bytes 3000000 --max-chars 80000
Options
| Flag |
Description |
-u, --url <url> |
Target URL to fetch (required) |
-o, --output <path> |
Write result JSON to a file |
--timeout <ms> |
Request timeout (default: 20000) |
--max-bytes <n> |
Max response bytes to read (default: 2000000) |
--max-chars <n> |
Max output text length (default: 50000) |
--max-redirects <n> |
Max redirects to follow (default: 4) |
--user-agent <value> |
Custom user agent |
--allow-private-network |
Allow private/internal targets (insecure, disabled by default) |
Security Defaults
modfetch blocks common SSRF targets by default:
localhost, loopback, and private-network IP ranges
- Cloud metadata endpoints (for example,
169.254.169.254, metadata hostnames)
- URL credentials (
https://user:pass@host)
- Excessive redirects
- Oversized responses or long-running requests
If you need to fetch trusted internal pages, use --allow-private-network explicitly.
Agent Skill
Notes
modfetch focuses on page fetching only.
modsearch and modlens belong to separate projects and are intentionally out of scope.
Disclaimer
This project is for personal learning and experimentation only. It is not intended for commercial use.
License
MIT