RobotsTxtParser
An extensible robots.txt parser and client library, with full support for every directive and specification.
- 24
- PHP
- MIT
- 6 y ago
An extensible robots.txt parser and client library, with full support for every directive and specification.
A pure-Python robots.txt parser with support for modern conventions.
NodeJS robots.txt parser with support for wildcard (*) matching.
:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API