๐ SubPDF ๐

๐ Discover Hidden Domains & Subdomains from PDFs Effortlessly!
SubPDF is a sleek, powerful, and user-friendly command-line tool designed to extract domains and subdomains from PDF files with lightning speed. Whether you're a cybersecurity analyst, data scientist, or simply managing extensive document repositories, SubPDF streamlines the process of uncovering valuable link information embedded within PDFs.
๐ Table of Contents
๐ Features
- โก Multi-threaded Fast Processing: Utilize multi-threading to handle numerous PDFs concurrently, drastically reducing processing time.
- ๐ Comprehensive Link Extraction: Extracts both domains and subdomains from clickable annotations and text within PDFs.
- ๐ Flexible Input Sources: Accepts single URLs, direct PDF links, input lists (text files), and JSON files containing URLs.
- ๐ฅ๏ธ Customizable Output: Choose from various output formats including JSON, simple lists, detailed reports, and more.
- ๐๏ธ Ephemeral & Permanent Storage: Option to store downloaded PDFs temporarily (auto-deleted after processing) or permanently.
- ๐ Custom HTTP Headers: Add custom headers to HTTP requests for enhanced compatibility and access.
๐ป Installation
1. Clone the Repository
git clone https://github.com/securezeron/SubPDF.git
cd SubPDF
2. Create a Virtual Environment (Recommended)
- Activate the Virtual Environment:
- Windows:
- Unix/Linux/MacOS:
3. Install Dependencies
pip install -r requirements.txt
Note: Ensure you have Python 3.7 or higher installed.
๐ ๏ธ Usage
SubPDF is a command-line tool. Below are instructions to help you get started seamlessly.
๐ Basic Commands
๐ง Advanced Options
๐ Help Command
For a comprehensive list of options and their descriptions, use:
๐ Output Formats
SubPDF offers multiple output formats to cater to different user needs:
-
Default (default):
- Description: Detailed report listing each PDF and its associated domains and subdomains.
- Usage:
python main.py -u https://www.example.com --format default
-
Simple (simple):
- Description: Line-by-line list of all extracted domains and subdomains without additional labeling.
- Usage:
python main.py -u https://www.example.com --format simple
-
JSON (json):
- Description: Structured JSON output mapping each PDF to its domains and subdomains.
- Usage:
python main.py -u https://www.example.com --format json --output-file results.json
-
List (list):
- Description: Bulleted list format for easy readability.
- Usage:
python main.py -u https://www.example.com --format list
-
Domains Only (domains):
- Description: Extracted registered domains using
tldextract, omitting subdomains for a cleaner overview.
- Usage:
python main.py -u https://www.example.com --format domains
๐ฅ Examples
1. Extract Domains from a Single Webpage
python main.py -u https://www.example.com
- Description: Crawls the specified webpage for PDF links, extracts domains and subdomains from each PDF, and displays the results in the default format.
2. Process Multiple PDFs from a List with Custom Headers
python main.py -il urls.txt -H "Authorization: Bearer YOUR_TOKEN" -H "Custom-Header: Value" --output-file domains.json
- Description: Reads URLs from
urls.txt, uses custom HTTP headers for requests, extracts domains and subdomains, and saves the results in domains.json.
3. Extract Domains in JSON Format with Debug Logs
python main.py -pu https://www.example.com/sample.pdf --format json --debug
- Description: Processes a direct PDF link, outputs the results in JSON format, and provides detailed debug logs for troubleshooting.
4. Limit Processing to 50 Threads and Store PDFs Permanently
python main.py -u https://www.example.com -t 50 -f ./downloads
- Description: Uses up to 50 threads for processing, stores downloaded PDFs in the
./downloads directory permanently.
๐ฆ Dependencies
SubPDF relies on the following Python libraries:
Installation via pip:
pip install -r requirements.txt
Contents of requirements.txt:
requests
beautifulsoup4
PyPDF2
tldextract
tqdm
๐ค Contributing
Contributions are the heart of SubPDF! Whether it's reporting a bug, suggesting a feature, or submitting a pull request, your input helps make SubPDF better for everyone.
1. Fork the Repository
Click the Fork button at the top right of the repository page.
2. Create a New Branch
git checkout -b feature/YourFeatureName
3. Make Your Changes
Implement your feature or fix the bug.
4. Commit Your Changes
git commit -m "Add a descriptive message about your changes"
5. Push to Your Fork
git push origin feature/YourFeatureName
6. Create a Pull Request
Navigate to the original repository and click on New Pull Request.
๐ License
Distributed under the MIT License.
Note: Ensure that you include a LICENSE file in your repository with the appropriate license text.
๐ Contact
Suman Chakraborty
Email: suman@zeron.one
LinkedIn: linkedin.com/in/suman-chakraborty-b857901b1
GitHub: github.com/suman-zeron
For any inquiries, issues, or feedback, feel free to reach out!
๐ Acknowledgements
Made with โค๏ธ by Suman@ZERON