A high-level Web Crawling and Web Scraping framework
Scrapy middleware for Zyte Smart Proxy Manager
A Scrapy middleware for scraping Wayback Machine snapshots from archive.org.
Scrapy middleware with wayback machine support for more robust scrapers.
A middleware to change user-agent in request for Scrapy
Automatically pick an User-Agent for every request
Scrapy middleware with TOR support for more robust scrapers or anonymous scraping.
Scrapy support for working with streamcorpus Stream Items