Topic: spider

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

A collection of awesome web crawler,spider in different languages

:paw_prints: Creeper - The Next Generation Crawler Framework (Go)

simple python spider frame, simple python crawler frame

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）

Free Web Scraping Tool

Creating Scrapy scrapers via the Django admin interface

爬虫集合

Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns