Topic: crawler

scrapy

scrapy

20635 0,17,20,36,26,0,21,0

Scrapy, a fast high-level web crawling & scraping framework for Python.

pyspider

pyspider

8872 0,9,14,0,9,12,0,12

A Powerful Spider(Web Crawler) System in Python.

webmagic

webmagic

4510 0,7,11,0,14,8,17,0

A scalable web crawler framework for Java.

node-crawler

node-crawler

2686 0,0,1,3,0,11,3,1

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

pholcus

pholcus

2362 0,19,4

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

scrapy-redis

scrapy-redis

2078 0,6,3,1,15

Redis-based components for Scrapy.

awesome-crawler

awesome-crawler

1739 0

A collection of awesome web crawler,spider in different languages

arachni

arachni

1436 0,1,8

Web Application Security Scanner Framework

gocrawl

gocrawl

1326 0,-1,1,0,2,0,8

Polite, slim and concurrent web crawler.

creeper

creeper

563

:paw_prints: Creeper - The Next Generation Crawler Framework (Go)

SwiftLinkPreview

SwiftLinkPreview

561 0,0

It makes a preview from an URL, grabbing all the information such as title, relevant texts and image

PSpider

PSpider

526

simple python spider frame, simple python crawler frame