html2rss-web
🕸 Turn almost any website into an RSS feed and serve it instantly. Automatic scraping. Start fast with Docker.
- 200
- Ruby
- MIT
- 2 d ago
Scraper API bridge between html2rss and Botasaurus for rendered-page fetching.
Repo html2rss/botasaurus-scrape-api, read from GitHub 9 h ago. html2rss has not joined Take My Repo.
html2rss/botasaurus-scrape-api is a public repo on GitHub by html2rss. Scraper API bridge between html2rss and Botasaurus for rendered-page fetching. It declares the MIT license on GitHub. It is written mainly in Python, has 1 stars and last changed on 2026-11-18. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-22.
In the maintainer's words. Shown from GitHub; Take My Repo does not keep a copy.
Docker-only FastAPI service that uses Botasaurus to fetch rendered HTML and best-effort response metadata.
GET /healthPOST /scrapeSCRAPE_MAX_WORKERS, default 4), with per-request total and post-boot work timeouts (SCRAPE_TIMEOUT_SECONDS, default 45; SCRAPE_WORK_TIMEOUT_SECONDS, default 30).SENTRY_DSN is set (see Sentry).curlpython3 (used by smoke assertions)The supported run is a hardened container: read-only root filesystem, all
capabilities dropped, no-new-privileges, and writable tmpfs mounts for /tmp
and $HOME (uid/gid of appuser). Chromium crashpad needs a writable HOME;
request profiles live under /tmp/scrape on the /tmp tmpfs.
docker run --rm -p 4010:4010 \
--init --shm-size=1gb \
--read-only --cap-drop ALL --security-opt no-new-privileges:true \
--tmpfs /tmp:size=512m \
--tmpfs /home/appuser:size=128m,uid=1000,gid=1000,mode=700 \
--memory 2g \
html2rss/botasaurus-scrape-api:latest
Or run from this repository directory (make serve uses the same hardening):
make serve
Health check:
make health
Example scrape:
make scrape-example
Use a dedicated Sentry project — map BOTASAURUS_SENTRY_DSN to SENTRY_DSN; do not reuse the html2rss-web DSN.
services:
botasaurus:
image: html2rss/botasaurus-scrape-api:latest
restart: unless-stopped
ports:
- "127.0.0.1:4010:4010"
environment:
SENTRY_DSN: ${BOTASAURUS_SENTRY_DSN:-}
SENTRY_ENVIRONMENT: ${ENVIRONMENT:-production}
SENTRY_RELEASE: ${GIT_SHA:-unknown}
Docker Hub image:
html2rss/botasaurus-scrape-api
Pull latest:
docker pull html2rss/botasaurus-scrape-api:latest
Pull immutable commit tag:
docker pull html2rss/botasaurus-scrape-api:<git-sha>
Publish policy:
main branch pushes.latest and the full commit SHA.Run end-to-end smoke checks (build, boot, health, scrape happy path, localhost guardrail, diagnostics, isolation). Smoke containers use the same production hardening flags as the supported docker run above and assert no /tmp/bota leftover after browser scrapes.
make smoke
Expected result: script prints [smoke] PASS and exits 0.
Machine-readable contract: openapi.yaml, generated from the FastAPI app with make openapi. Do not hand-edit it. make check runs make openapi-verify and fails if the snapshot does not match a fresh app.openapi() dump.
A running container also serves FastAPI's live schema and UIs at /openapi.json, /docs, and /redoc. Those are framework defaults, not additional scrape endpoints.
Human examples follow. OpenAPI 2.0.0 is a breaking wire cut from 1.x:
scroll_to_bottom → scroll (scroll means scroll-to-bottom / lazy-load)window_size: [w, h] → { "width": w, "height": h }blocked_detected / challenge_detected / detected_challenge → diagnostics.challenge.{blocked,detected,marker}request_id, attempts, strategy_used, render_ms, execution_tier, optional timeout_phase → diagnostics.*html; 400/422 error_category is validationScrapeResponse → ScrapeSuccess + ScrapeErrorGET /healthReturns service status and detected Botasaurus version.
Example shape:
{
"status": "ok",
"service": "botasaurus-scrape-api",
"botasaurus_version": "4.x.x"
}
POST /scrapeOptional transport header X-Request-Id: when present and path-safe, the same value is echoed in diagnostics.request_id on every response envelope (success and error).
Request body (minimum):
{
"url": "https://example.com"
}
Request body (full options):
{
"url": "https://example.com",
"execution_mode": "auto",
"navigation_mode": "auto",
"max_retries": 2,
"wait_for_selector": "h1",
"wait_timeout_seconds": 15,
"scroll": true,
"block_images": true,
"block_images_and_css": false,
"block_trackers": true,
"wait_for_complete_page_load": true,
"user_agent": "Mozilla/5.0 ...",
"headers": {
"Accept-Language": "en-US,en;q=0.9",
"Cookie": "session=..."
},
"cookies": {
"session": "..."
},
"window_size": {"width": 1920, "height": 1080},
"lang": "en-US",
"headless": false,
"proxy": "http://user:pass@proxy:port"
}
Request options (contract):
execution_mode:auto (default): attempts fast anti-detect HTTP request (curl_cffi / TLS fingerprinting) first; escalates to real browser driver if challenge or dynamic hydration is required.request: anti-detect HTTP request tier only (fastest, low memory).browser: full headless/stealth Chromium browser tier.navigation_mode:auto (default): google_get -> google_get(bypass_cloudflare=true) -> getget: only getgoogle_get: only google_getgoogle_get_bypass: only google_get(bypass_cloudflare=true)max_retries: 0..3, default 2 (attempts = 1 + max_retries, with auto capped by 3 strategy steps).wait_for_selector: if set, response waits for selector before capture (routes to browser tier).wait_timeout_seconds: selector wait timeout (default 15). Values outside [1, SCRAPE_WORK_TIMEOUT_SECONDS] (default 30) are clamped into that range so scrape still runs.scroll: if true, scrolls to the bottom to trigger lazy-loaded feeds (routes to browser tier).block_images: pass image blocking to driver. Default true.block_images_and_css: pass image+css blocking to driver. Default false.block_trackers: block tracking/ad networks and web fonts to speed up rendering. Default true.wait_for_complete_page_load: pass page-load wait behavior to driver. Default true.user_agent: explicit user agent string passed to driver.headers: custom HTTP request headers forwarded to request client or browser session.cookies: key-value cookies map forwarded to request client or browser session.window_size: viewport object { "width": 1920, "height": 1080 } passed to driver.lang: browser language passed to driver (for example en-US).headless: pass headless browser mode to driver. Default false.proxy: proxy URL passed to driver. Invalid or blocked proxy URLs are rejected by SSRF guardrails.Success response (ScrapeSuccess, HTTP 200):
{
"url": "https://example.com",
"final_url": "https://example.com/",
"status_code": 200,
"headers": {
"content-type": "text/html; charset=utf-8"
},
"html": "<!doctype html>...",
"metadata_error": null,
"xhr_responses": [],
"diagnostics": {
"request_id": "b01ef2f8-f641-4e75-8ef2-0b73f7b4f372",
"attempts": 1,
"strategy_used": null,
"render_ms": 154,
"execution_tier": "http_request",
"challenge": {
"blocked": false,
"detected": false,
"marker": null
}
}
}
Error response (ScrapeError, HTTP 400/403/422/502/504). No html:
{
"url": "https://example.com",
"error": "Target URL is blocked",
"error_category": "validation",
"diagnostics": {
"request_id": "b01ef2f8-f641-4e75-8ef2-0b73f7b4f372",
"attempts": 0,
"strategy_used": null,
"render_ms": 0,
"execution_tier": null,
"challenge": null
}
}
Field behavior:
html: rendered page HTML, UTF-8-normalized. Present on success only.headers, status_code, final_url: best-effort metadata and may be null. When html is present, document headers content-type is text/html; charset=utf-8.error: failure message on ScrapeError.metadata_error: populated when metadata extraction fails but HTML scrape succeeds.cookies: always-on additive name/value map of cookies set for the target host during the scrape (e.g. an anti-bot clearance cookie). Send it back in the cookies request field to reuse the session; the service itself stays stateless. Success responses only.diagnostics.request_id: unique per request for tracing.diagnostics.attempts: actual attempts performed.diagnostics.strategy_used: browser navigation strategy on the final attempt, or null on the HTTP-request tier.diagnostics.render_ms: elapsed render/runtime milliseconds.diagnostics.execution_tier: http_request or browser_driver.diagnostics.challenge: anti-bot assessment (blocked, detected, marker), or null when detection did not run.diagnostics.timeout_phase: on error_category=timeout only — which stage burned the budget: queue (threadpool wait), boot (browser/driver start), or work (navigate/wait/scroll). null on non-timeout outcomes.xhr_responses: always-on additive list of JSON XHR/fetch sub-resource bodies captured during the browser tier (empty for HTTP-request tier). Each entry is {url, status_code, headers, body}. headers keep only content-type (Set-Cookie and other headers are dropped). Caps: at most 20 responses, 500 KB per body, 2 MB aggregate across bodies. Main document responses are excluded. Collector state is reset between strategy retries so challenge interstitials do not pollute a later successful attempt. Candidate filtering for article-likeness is a client concern.error_category:timeout — outer handler budget, queue wait, or a tier exception whose message contains timeout (browser and request tiers). Those tier timeouts also set diagnostics.timeout_phase (boot/work; request-tier timeouts are work).challenge_blocknavigation_errormetadata_errorvalidation (400 URL rejection and 422 schema failures)Status codes:
200: scrape completed (ScrapeSuccess).400: URL rejected by validation (for example unresolved host). error_category is validation.403: URL blocked by SSRF guardrails.422: request schema validation failed. Body is the scrape error envelope (url, error, error_category, diagnostics), not FastAPI {"detail":[...]}. error_category is validation.502: scrape execution failure/challenge block.504: scrape timed out.| Signal | Ops meaning |
|---|---|
timeout_phase=queue |
Capacity/workers — threadpool wait burned the budget |
timeout_phase=boot |
RAM/shm/prewarm canary — browser/driver failed to start in time |
timeout_phase=work |
Slow or hostile target — navigate/wait/scroll exceeded budget |
challenge_block |
Product signal — anti-bot surface (502), not a timeout |
On outer 504, queued Futures are cancelled (work never starts). In-flight scrapes stop at the next deadline check or step timeout.
POST /scrape executes this path:
loop.run_in_executor)./tmp/scrape/<request_id>.Driver(...) with request options.auto or explicit mode) and optional selector wait.finally:Enforced invariants:
metadata_error is set instead).The service accepts only http and https input URLs and blocks sensitive destinations before scrape execution.
Blocked targets include:
localhost and *.localhostException:
64:ff9b::/96 is allowed./scrape request gets its own runtime directory: /tmp/scrape/<request_id>.finally: driver close + runtime directory delete + request-id in-memory state scrub./tmp/scrape on tmpfs in production (see html2rss-web docker-compose.yml).Use a separate Sentry project from html2rss-web (BOTASAURUS_SENTRY_DSN → SENTRY_DSN). Disabled when SENTRY_DSN is unset.
| Variable | Default | Description |
|---|---|---|
SENTRY_DSN |
(unset) | Project DSN (Settings.sentry.dsn). |
SENTRY_ENVIRONMENT |
production |
Deployment tag (Settings.sentry.environment; ENVIRONMENT fallback). |
SENTRY_RELEASE |
image git SHA | Release tag on events (Settings.sentry.release). Published images bake the commit SHA via GIT_SHA build-arg; override at runtime if needed. |
SENTRY_SEND_DEFAULT_PII |
false |
Send default PII when true (Settings.sentry.send_default_pii). |
Signal routing: Sentry is error tracking only (no metrics, tracing or profiling). Only service-side failures become grouped Issues: queue/boot timeouts and failures before any navigation attempt (browser boot, storage). Target-site outcomes (challenge_block, work timeouts, navigation errors on the page) are product signals: query them from the structured scrape_finished log event in your log pipeline.
| Variable | Default | Description |
|---|---|---|
LOG_FORMAT |
text |
Log line format. text keeps event k=v messages (smoke greps). json emits one JSON object per line with ts, level, logger, event_name, msg, and typed fields for collectors. Each admitted scrape emits one terminal scrape_finished event (request_id, host, modes, tier, status, error_category, timeout_phase, attempts, warm_hit, cookies_sent, proxy_configured, marker, vendor, signal, render_ms, duration_ms). vendor (datadome, cloudflare, vercel, unknown) and signal (html, header, status, driver, surface) are set on challenge_block only. request_tier_escalating and scrape_challenge_detected carry the same marker/vendor/signal fields. |
SCRAPE_MAX_WORKERS |
4 |
Threadpool worker limit for sync browser execution. On low-RAM hosts (for example Docker with --memory=768m or a 1–2 vCPU VM), use 1 or 2 to limit concurrent Chromium boots; higher values increase queue wait and swap pressure without improving wall time. |
SCRAPE_TIMEOUT_SECONDS |
45 |
Handler wall-clock budget in seconds (queue, browser boot, and work). |
SCRAPE_WORK_TIMEOUT_SECONDS |
30 |
Post-boot navigate, selector wait, and scroll budget in seconds. |
SCRAPE_RUNTIME_MIN_FREE_BYTES |
268435456 (256 MiB) |
Prune orphan runtime dirs when free space drops below this threshold. |
SCRAPE_PREWARM |
false |
Opt-in single-slot Chromium prewarm. When true, fingerprint-matching browser scrapes reuse a spare driver built after the previous scrape. Requires single uvicorn worker when enabled; multi-worker deployments do not share spare state and worker restarts delete foreign spare-* dirs. Keep off until canary on --shm-size=1gb + init: true shows stable RSS. |
SCRAPE_PREWARM_IDLE_TTL_SECONDS |
600 |
Close an unused warm spare after this many seconds. 0 disables idle reap — one Chromium remains in RAM until process exit. |
SCRAPE_PREWARM_MIN_REFILL_SECONDS |
30 |
Minimum interval between spare rebuilds. |
SCRAPE_RUNTIME_STATS_INTERVAL_SECONDS |
60 |
Emit a runtime_stats heartbeat (cgroup memory, Chromium process count, /tmp free bytes, runtime dirs, active requests, warm spare present). 0 disables the heartbeat. Startup always runs runtime_preflight (HOME writable, runtime root writable, Chromium binary present). |
Easy mode:
curl -s -X POST http://localhost:4010/scrape \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com"}'
Hard-target mode:
curl -s -X POST http://localhost:4010/scrape \
-H 'Content-Type: application/json' \
-d '{
"url":"https://truthsocial.com/@realDonaldTrump",
"navigation_mode":"auto",
"max_retries":2,
"wait_timeout_seconds":15,
"headless":false
}'
Challenge-target mode (recommended):
curl -s -X POST http://localhost:4010/scrape \
-H 'Content-Type: application/json' \
-d '{
"url":"https://www.wsj.com/",
"navigation_mode":"auto",
"max_retries":2,
"headless":false,
"proxy":"http://user:pass@residential-proxy:port"
}'
Note: if your IP is already flagged, you may still get challenge pages. In that case use a fresh residential IP.
make / make check: Ruff, Hadolint, Spectral, unit tests, and make openapi-verify.make lint: Ruff plus Hadolint plus Spectral.make spectral: lint openapi.yaml with Spectral in Docker (stoplight/spectral:6).make ready: same as make check (pre-PR gate; run make smoke when Docker behavior changes).make test: run host unit tests.make openapi: regenerate openapi.yaml from app.openapi().make openapi-verify: fail if openapi.yaml does not match a fresh dump.make build: build Docker image.make serve: build and run API container on localhost:4010.make health: call GET /health on running service.make scrape-example: call POST /scrape with https://example.com.make smoke: run end-to-end smoke suite.html2rss/botasaurus-scrape-api is a public repo on GitHub by html2rss. Scraper API bridge between html2rss and Botasaurus for rendered-page fetching.
It declares the MIT license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2026-11-18. Take My Repo does not judge from that whether it is maintained.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
🕸 Turn almost any website into an RSS feed and serve it instantly. Automatic scraping. Start fast with Docker.
📰 Build RSS 2.0 feeds from websites (and JSON APIs) automatically or with a few CSS selectors.
Our website. Serves project documentation and showcases out of the box available feeds.
This page is also available as structured data, with the same sources and dates.
/api/v1/resources/gh/html2rss/botasaurus-scrape-apiTake My Repo has not tested this on your machine. Requirements come from the project's own documentation.