pholcus
2362
0,19,4
[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
Anti-Anti-Spider
508
越来越多的网站具有反爬虫特性,有的用图片隐藏关键数据,有的使用反人类的验证码,建立反反爬虫的代码仓库,通过与不同特性的网站做斗争(无恶意)提高技术。(欢迎提交难以采集的网站)
Crawler-Detect
235
0,1
CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
grab-site
176
0
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
- «
- »