Developed by devhub.io
Scrapy, a fast high-level web crawling & scraping framework for Python.
A Powerful Spider(Web Crawler) System in Python.
A scalable web crawler framework for Java.
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
Redis-based components for Scrapy.
A collection of awesome web crawler,spider in different languages
Web Application Security Scanner Framework
Polite, slim and concurrent web crawler.
:paw_prints: Creeper - The Next Generation Crawler Framework (Go)
It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.
simple python spider frame, simple python crawler frame