ScrapeHubAI
๐ Powered by ScrapeGraphAI - The most advanced AI-powered web scraping API
An open-source LangGraph-based agent that analyzes GitHub stargazers, traces them to their companies, and evaluates these companies as potential sales targets for AI scraping infrastructure.
๐ Features
- GitHub Stargazer Analysis: Fetches and analyzes users who starred a repository
- Company Identification: Traces GitHub users to their affiliated companies
- Intelligent Evaluation: Scores companies based on size, industry, and technology fit
- Web Scraping: Uses ScrapeGraphAI API to gather additional company information
- Beautiful UI: Streamlit-based interface for easy interaction
- Export Results: Download analysis results as CSV for further processing
๐ Quick Start
Prerequisites
- Python 3.10 or higher
- API keys for GitHub, OpenRouter, and ScrapeGraphAI
Installation
- Clone the repository:
git clone https://github.com/yourusername/ScrapeHubAI.git
cd ScrapeHubAI
- Install dependencies:
pip install -r requirements.txt
- Configure API keys:
Getting a GitHub Personal Access Token
- Go to GitHub โ Settings โ Developer settings โ Personal access tokens โ Tokens (classic)
- Click "Generate new token" โ "Generate new token (classic)"
- Give your token a descriptive name (e.g., "ScrapeHub")
- Select the following scopes:
public_repo (required for reading public repositories)
read:user (optional, for better user information)
read:org (optional, for organization information)
- Click "Generate token" at the bottom
- Important: Copy your token immediately - you won't be able to see it again!
Setting up your .env file
Create a .env file in the project root:
GITHUB_TOKEN=your_github_personal_access_token
OPENROUTER_API_KEY=your_openrouter_api_key
OPENROUTER_API_BASE=https://openrouter.ai/api/v1
SGAI_API_KEY=your_scrapegraphai_api_key
Or copy from the example:
cp .env.example .env
# Then edit .env with your actual API keys
Getting Other API Keys
- Run the application:
๐ How It Works
- Fetch Stargazers: The agent retrieves users who starred the specified GitHub repository
- Trace to Companies: Identifies companies through user profiles and organization memberships
- Gather Intelligence: Uses ScrapeGraphAI to scrape additional company information
- Evaluate & Rank: Scores companies based on multiple criteria:
- Technology relevance (AI, ML, data analytics, scraping)
- Industry fit (e-commerce, SaaS, fintech)
- Company size and growth indicators
- Explicit data processing needs
๐ฏ Use Cases
- Sales Intelligence: Identify potential customers for AI/data infrastructure products
- Market Research: Understand which companies are interested in specific technologies
- Partnership Discovery: Find companies with complementary technology needs
- Competitive Analysis: See which companies are following competitor repositories
๐ ๏ธ Architecture
- LangGraph: Orchestrates the multi-step analysis workflow
- OpenRouter: Provides LLM capabilities for intelligent evaluation
- ScrapeGraphAI: Powers web scraping for company information
- Streamlit: Creates an intuitive user interface
๐ Project Structure
github-star-to-company-agent/
โโโ src/
โ โโโ agent.py # LangGraph workflow definition
โ โโโ tools.py # GitHub and scraping tools
โ โโโ evaluator.py # Company scoring logic
โ โโโ app.py # Streamlit UI
โโโ tests/
โ โโโ test_agent.py # Unit tests
โโโ docs/
โ โโโ usage.md # Detailed usage guide
โโโ .env # API keys (create this)
โโโ requirements.txt # Python dependencies
โโโ README.md # This file
๐งช Testing
Run the test suite:
๐ง Configuration
Environment Variables
GITHUB_TOKEN: Personal access token for GitHub API
OPENROUTER_API_KEY: API key for OpenRouter LLM service
OPENROUTER_API_BASE: OpenRouter API endpoint (default: https://openrouter.ai/api/v1)
SGAI_API_KEY: API key for ScrapeGraphAI service
Advanced Settings
Customize analysis parameters through the Streamlit UI:
- Maximum number of companies to display
- Minimum score threshold for filtering
- Analysis depth and timeout settings
โ๏ธ Ethical Considerations
- Respect Rate Limits: The agent implements automatic rate limiting
- Privacy: Only analyzes publicly available information
- Terms of Service: Ensure compliance with all platform ToS
- Responsible Use: Designed for legitimate business intelligence only
๐ค Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
๐ License
This project is licensed under the MIT License - see the LICENSE file for details.
๐ Acknowledgments
๐ Support
For issues, questions, or suggestions:
- Check the usage guide
- Open an issue on GitHub
- Review existing issues for solutions
Made with โค๏ธ from ScrapeGraphAI team