crawling

Open repos and models their makers tagged crawling, most starred first. The same for everyone.

AI agentsRepo

Scrapling

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

  • 104k
  • Python
  • BSD-3-Clause
  • 1 d ago
D4VinciTake it
AutomationRepo

crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  • 26k
  • TypeScript
  • Apache-2.0
  • 1 h ago
apifyTake it
AutomationRepo

crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

  • 9.7k
  • Python
  • Apache-2.0
  • 2 h ago
apifyTake it
Search GitHub and Hugging Face for crawling

Get , open it once and sign in. Then come back here and press the button again.

crawling: open source repos and models on Take My Repo