trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- 7.2k
- Python
- Apache-2.0
- 11 h ago
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
Multi-Language Dataset Cleaner/Creator for Mozilla's DeepSpeech Framework
Praaline is an open-source system to manage, annotate, visualise and analyse spoken language corpora
Article title, authors, date and body extraction dataset.
A Scrapy package based web scraper for collecting Kurdish text data from websites. The tool recursively crawls specified domains, extracts article content using Trafilatura, and filters results by language using Facebook's FastText language identification model.
Recon is a Java-based tool for the annotation of relations among textual elements and semantic concepts.