kiddyuchina / Beanbun
Licence: mit
Beanbun 是用 PHP 编写的多进程网络爬虫框架,具有良好的开放性、高可扩展性,基于 Workerman。
Stars: ✭ 1,096
Projects that are alternatives of or similar to Beanbun
Awesome Python Primer
自学入门 Python 优质中文资源索引,包含 书籍 / 文档 / 视频,适用于 爬虫 / Web / 数据分析 / 机器学习 方向
Stars: ✭ 57 (-94.8%)
Mutual labels: crawler, spider
Gospider
Gospider - Fast web spider written in Go
Stars: ✭ 785 (-28.38%)
Mutual labels: crawler, spider
Grab Site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Stars: ✭ 680 (-37.96%)
Mutual labels: crawler, spider
Crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Stars: ✭ 8,392 (+665.69%)
Mutual labels: crawler, spider
Creeper
🐾 Creeper - The Next Generation Crawler Framework (Go)
Stars: ✭ 762 (-30.47%)
Mutual labels: crawler, spider
Zhihu Crawler
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Stars: ✭ 890 (-18.8%)
Mutual labels: crawler, spider
Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (-40.15%)
Mutual labels: crawler, spider
Photon
Incredibly fast crawler designed for OSINT.
Stars: ✭ 8,332 (+660.22%)
Mutual labels: crawler, spider
Icrawler
A multi-thread crawler framework with many builtin image crawlers provided.
Stars: ✭ 629 (-42.61%)
Mutual labels: crawler, spider
Crawler
A high performance web crawler in Elixir.
Stars: ✭ 781 (-28.74%)
Mutual labels: crawler, spider
Netdiscovery
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Stars: ✭ 573 (-47.72%)
Mutual labels: crawler, spider
Douyin
API of DouYin for Humans used to Crawl Popular Videos and Musics
Stars: ✭ 580 (-47.08%)
Mutual labels: crawler, spider
Nodespider
[DEPRECATED] Simple, flexible, delightful web crawler/spider package
Stars: ✭ 33 (-96.99%)
Mutual labels: crawler, spider
Avbook
AV 电影管理系统, avmoo , javbus , javlibrary 爬虫,线上 AV 影片图书馆,AV 磁力链接数据库,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database
Stars: ✭ 8,133 (+642.06%)
Mutual labels: crawler, spider
简介
Beanbun 是一个简单可扩展的爬虫框架,支持分布式,支持守护进程模式与普通模式,守护进程模式基于 Workerman,下载器基于 Guzzle。
文档
https://github.com/kiddyuchina/Beanbun/blob/master/docs/chs/README.md
特点
- 支持守护进程与普通两种模式(守护进程模式只支持 Linux 服务器)
- 默认使用 guzzle 进行爬取
- 支持分布式
- 支持内存、Redis 等多种队列方式
- 支持自定义URI过滤
- 支持广度优先和深度优先两种爬取方式
- 遵循 PSR-4 标准
- 爬取网页分为多步,每步均支持自定义动作(如添加代理、修改 user-agent 等)
- 灵活的扩展机制,可方便的为框架制作插件:自定义队列、自定义爬取方式...
安装
Beanbun 可以通过 composer 进行安装。
$ composer require kiddyu/beanbun
快速开始
创建一个文件 start.php,包含以下内容
<?php
use Beanbun\Beanbun;
$beanbun = new Beanbun;
$beanbun->seed = [
'http://www.950d.com/',
'http://www.950d.com/list-1.html',
'http://www.950d.com/list-2.html',
];
$beanbun->afterDownloadPage = function($beanbun) {
file_put_contents(__DIR__ . '/' . md5($beanbun->url), $beanbun->page);
};
$beanbun->start();
在命令行中执行
$ php start.php
接下来就可以看到抓取的日志了。
插件
- beanbun-parser 数据抽取插件 https://github.com/kiddyuchina/beanbun-parser
更多详细内容,请查看 文档
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].