awesome-seo
Curated SEO, GEO & AI Search resources — technical SEO, agent discoverability, analytics, and traffic monetization.
- 2.9k
- MIT
- 5 wk ago
An extensible robots.txt parser and client library, with full support for every directive and specification.
Repo VIPnytt/RobotsTxtParser, read from GitHub 2 min ago. VIPnytt has not joined Take My Repo.
VIPnytt/RobotsTxtParser is a public repo on GitHub by VIPnytt. An extensible robots.txt parser and client library, with full support for every directive and specification. It declares the MIT license on GitHub. It is written mainly in PHP, has 24 stars and last changed on 2021-04-10. Nobody has reviewed it on Take My Repo yet, which says nothing about its quality. Read from GitHub on 2026-11-22.
VIPnytt/RobotsTxtParser is a public repo on GitHub by VIPnytt. An extensible robots.txt parser and client library, with full support for every directive and specification.
It declares the MIT license on GitHub.
Its maker has not tagged a system, so it is unknown here. Its README says what it needs.
Open "Take it" on its page: Take My Repo writes an install plan for your machine and hands it to your AI agent (Claude Code, Codex or Cursor), which asks you before each step. Take My Repo has not tested it on any machine.
Its last change on GitHub was on 2021-04-10. GitHub counts 1 open issue and pull requests. Take My Repo does not judge from that whether it is maintained.
Nobody has reviewed it on Take My Repo yet, which says nothing about its quality.
NodeJS robots.txt parser with support for wildcard (*) matching.
🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard
A pure-Python robots.txt parser with support for modern conventions.
Open-source SEO & GEO audit: a zero-dependency CLI (npx), an agent skill for Claude Code, Cursor, Codex, Gemini & Copilot, 47 SEO prompts and a cited Google/Bing/AI-search knowledge base. Checks robots.txt, sitemaps, canonicals, hreflang, structured data, Core Web Vitals and AI crawlers — then fixes them.