ssssssss-team / Spider Flow
Licence: mit
新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。
Stars: ✭ 365
Programming Languages
java
68154 projects - #9 most used programming language
Projects that are alternatives of or similar to Spider Flow
Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (+79.73%)
Mutual labels: crawler, spider, web-crawler
Crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Stars: ✭ 8,392 (+2199.18%)
Mutual labels: crawler, spider, web-crawler
Awesome Crawler
A collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+1213.15%)
Mutual labels: crawler, spider, web-crawler
Gopa
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (-24.11%)
Mutual labels: crawler, spider, web-crawler
Abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+437.26%)
Mutual labels: crawler, spider, web-crawler
Pspider
简单易用的Python爬虫框架，QQ交流群：597510560
Stars: ✭ 1,611 (+341.37%)
Mutual labels: crawler, spider, web-crawler
Maman
Rust Web Crawler saving pages on Redis
Stars: ✭ 39 (-89.32%)
Mutual labels: crawler, spider, web-crawler
crawler
A simple and flexible web crawler framework for java.
Stars: ✭ 20 (-94.52%)
Mutual labels: crawler, spider, jsoup
Crawlerforreader
Android 本地网络小说爬虫，基于jsoup及xpath
Stars: ✭ 312 (-14.52%)
Mutual labels: crawler, xpath, jsoup
Crawlab Lite
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122 (-66.58%)
Mutual labels: crawler, spider, web-crawler
Zhihu Crawler People
A simple distributed crawler for zhihu && data analysis
Stars: ✭ 182 (-50.14%)
Mutual labels: crawler, spider, web-crawler
flink-crawler
Continuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (-86.85%)
Mutual labels: crawler, spider, web-crawler
91porn Api
🌭💦 91porn爬虫在线无限制API接口（永久有效，口令每日更新） 及 在线web预览
Stars: ✭ 341 (-6.58%)
Mutual labels: crawler, spider
Spidy
The simple, easy to use command line web crawler.
Stars: ✭ 257 (-29.59%)
Mutual labels: crawler, web-crawler
galer
A fast tool to fetch URLs from HTML attributes by crawl-in.
Stars: ✭ 138 (-62.19%)
Mutual labels: crawler, spider
Hacker News Digest
📰 A responsive interface of Hacker News with summaries and thumbnails.
Stars: ✭ 278 (-23.84%)
Mutual labels: crawler, spider
Crawlertutorial
爬蟲極簡教學（fetch, parse, search, multiprocessing, API）- PTT 為例
Stars: ✭ 282 (-22.74%)
Mutual labels: crawler, spider
Weixin Spider
微信公众号爬虫，公众号历史文章，文章评论，文章阅读及在看数据，可视化web页面，可部署于Windows服务器。基于Python3之flask/mysql/redis/mitmproxy/pywin32等实现，高效微信爬虫，微信公众号爬虫，历史文章，文章评论，数据更新。
Stars: ✭ 287 (-21.37%)
Mutual labels: crawler, spider
Freshonions Torscraper
Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion
Stars: ✭ 348 (-4.66%)
Mutual labels: crawler, spider
介绍 | 特性 | 插件 | DEMO站点 | 文档 | 更新日志 | 截图 | 其它开源 | 免责声明
介绍
平台以流程图的方式定义爬虫,是一个高度灵活可配置的爬虫平台
特性
- [x] 支持Xpath/JsonPath/css选择器/正则提取/混搭提取
- [x] 支持JSON/XML/二进制格式
- [x] 支持多数据源、SQL select/selectInt/selectOne/insert/update/delete
- [x] 支持爬取JS动态渲染(或ajax)的页面
- [x] 支持代理
- [x] 支持自动保存至数据库/文件
- [x] 常用字符串、日期、文件、加解密等函数
- [x] 支持插件扩展(自定义执行器，自定义方法）
- [x] 任务监控,任务日志
- [x] 支持HTTP接口
- [x] 支持Cookie自动管理
- [x] 支持自定义函数
插件
项目部分截图
爬虫列表
爬虫测试
Debug
日志
其它开源项目
免责声明
请勿将spider-flow应用到任何可能会违反法律规定和道德约束的工作中,请友善使用spider-flow，遵守蜘蛛协议，不要将spider-flow用于任何非法用途。如您选择使用spider-flow即代表您遵守此协议，作者不承担任何由于您违反此协议带来任何的法律风险和损失，一切后果由您承担。
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].