crawlers
crawler
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
- 205
- Java
- Apache-2.0
- 2 d ago
agent-traffic
Tell agents, AI crawlers, search crawlers and bots apart from people
- 84
- TypeScript
- MIT
- 11 d ago
cloudflare-bot-directory
Cloudflare Radar verified bots directory — 500+ crawlers and search-engine user agents as data.
- 21
- JavaScript
- 6 d ago
born2crawl
A highly performant and versatile crawling engine, designed with scalability and extensibility in mind.
- 8
- Kotlin
- Apache-2.0
- 3 y ago
dezcrwl
🕷️ | dezcrwl is a website history crawler gather hidden information and check vulnerabilities for extracted .js endpoints & much more!
- 7
- Python
- MIT
- 21 mo ago
immoscout24-scraper
Immoscout24 Scraper — free, open-source Immoscout24 scraper. Unlimited German & Austrian listings, prices, photos, phones as clean JSON, no API key, no proxies.
- 1
- HTML
- MIT
- 4 d ago
gpt-data-exposure
An In-Depth Investigation of Data Collection in LLM App Ecosystems
- 4
- Python
- MIT
- 16 mo ago

