scrapy-plugins
Plugins for the Scrapy framework
Built from scrapy-plugins's public GitHub profile, The Internet. Not on Take My Repo yet, so there are no posts or updates here.
Scrapy extension to write scraped items using Django models
Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy
Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls
Scrapy middleware to add extra fields to items, like timestamp, response fields, spider attributes etc.
Scrapy schema validation pipeline and Item builder using JSON Schema
A scrapy extension to store requests and responses information in storage service
Scrapy spider middleware to clean up query parameters in request URLs
Scrapy spider middleware to split an item into multiple items using a multi-valued key
A scrapy extension to sync `.scrapy` folder to an S3 bucket
Snowflake database loading utility with Scrapy integration
Scrapy spider middleware to use Scrapinghub's Hub Crawl Frontier as a backend for URLs