web-crawling

Open repos and models their makers tagged web-crawling, most starred first. The same for everyone.

AI agentsRepo

crawl4ai

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

  • 88k
  • Python
  • Apache-2.0
  • 1 d ago
unclecodeTake it
AutomationRepo

crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  • 27k
  • TypeScript
  • Apache-2.0
  • 20 h ago
apifyTake it
AutomationRepo

crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

  • 9.7k
  • Python
  • Apache-2.0
  • 10 h ago
apifyTake it
GitHubRepo

courlan

Filter, clean, and sample URLs for web crawling, with language, spam, and tracker filters, link extraction, and a URL store for crawl scheduling. Python and CLI

  • 181
  • Python
  • Apache-2.0
  • 4 d ago
adbarTake it
AutomationRepo

apify-sdk-python

Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.

  • 180
  • Python
  • Apache-2.0
  • 2 d ago
apifyTake it
AutomationRepo

apify-client-python

Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.

  • 105
  • Python
  • Apache-2.0
  • 8 h ago
apifyTake it
AI agentsRepo

crawl4ai-mcp-server

🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted and free. Perfect for integrating web scraping into your AI workflows with OpenAI Agents SDK, Cursor, Claude Code, and other MCP-compatible tools.

  • 118
  • Python
  • MIT
  • 10 mo ago
sadiuysalTake it
Search GitHub and Hugging Face for web-crawling

Get , open it once and sign in. Then come back here and press the button again.

web-crawling: open source repos and models on Take My Repo