Topic: spider

node-crawler

node-crawler

2686 0,0,1,3,0,11,3,1

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

pholcus

pholcus

2362 0,19,4

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

awesome-crawler

awesome-crawler

1739 0

A collection of awesome web crawler,spider in different languages

creeper

creeper

563

:paw_prints: Creeper - The Next Generation Crawler Framework (Go)

PSpider

PSpider

526

simple python spider frame, simple python crawler frame

Anti-Anti-Spider

Anti-Anti-Spider

508

越来越多的网站具有反爬虫特性,有的用图片隐藏关键数据,有的使用反人类的验证码,建立反反爬虫的代码仓库,通过与不同特性的网站做斗争(无恶意)提高技术。(欢迎提交难以采集的网站)

newcrawler

newcrawler

474

Free Web Scraping Tool

django-dynamic-scraper

django-dynamic-scraper

456 0,1

Creating Scrapy scrapers via the Django admin interface

awesome-spider

awesome-spider

426

爬虫集合

antcolony

antcolony

280

Nodejs实现的一个磁力链接爬虫 http://findit.keenwon.com (原域名http://findit.so )

Crawler-Detect

Crawler-Detect

235 0,1

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

grab-site

grab-site

176 0

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns