crawlab-team / Crawlab Lite
Licence: bsd-3-clause
Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台
Stars: ✭ 122
Projects that are alternatives of or similar to Crawlab Lite
Crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Stars: ✭ 8,392 (+6778.69%)
Mutual labels: platform, crawler, spider, scrapy, web-crawler
Gopa
[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn
Stars: ✭ 277 (+127.05%)
Mutual labels: crawler, spider, web-crawler
flink-crawler
Continuous scalable web crawler built on top of Flink and crawler-commons
Stars: ✭ 48 (-60.66%)
Mutual labels: crawler, spider, web-crawler
Pspider
简单易用的Python爬虫框架,QQ交流群:597510560
Stars: ✭ 1,611 (+1220.49%)
Mutual labels: crawler, spider, web-crawler
Zhihu Crawler People
A simple distributed crawler for zhihu && data analysis
Stars: ✭ 182 (+49.18%)
Mutual labels: crawler, spider, web-crawler
Marmot
💐Marmot | Web Crawler/HTTP protocol Download Package 🐭
Stars: ✭ 186 (+52.46%)
Mutual labels: crawler, spider, scrapy
Awesome Crawler
A collection of awesome web crawler,spider in different languages
Stars: ✭ 4,793 (+3828.69%)
Mutual labels: crawler, spider, web-crawler
Awesome Web Scraper
A collection of awesome web scaper, crawler.
Stars: ✭ 147 (+20.49%)
Mutual labels: spider, scrapy, web-crawler
Icrawler
A multi-thread crawler framework with many builtin image crawlers provided.
Stars: ✭ 629 (+415.57%)
Mutual labels: crawler, spider, scrapy
Spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Stars: ✭ 656 (+437.7%)
Mutual labels: crawler, spider, web-crawler
Scrapingoutsourcing
ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个
Stars: ✭ 164 (+34.43%)
Mutual labels: crawler, spider, scrapy
Abot
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Stars: ✭ 1,961 (+1507.38%)
Mutual labels: crawler, spider, web-crawler
Goribot
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
Stars: ✭ 190 (+55.74%)
Mutual labels: crawler, spider, scrapy
Python3 Spider
Python爬虫实战 - 模拟登陆各大网站 包含但不限于:滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝,如果喜欢请start ❤️
Stars: ✭ 2,129 (+1645.08%)
Mutual labels: crawler, spider, scrapy
Spider Flow
新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。
Stars: ✭ 365 (+199.18%)
Mutual labels: crawler, spider, web-crawler
Haipproxy
💖 High available distributed ip proxy pool, powerd by Scrapy and Redis
Stars: ✭ 4,993 (+3992.62%)
Mutual labels: crawler, spider, scrapy
Maman
Rust Web Crawler saving pages on Redis
Stars: ✭ 39 (-68.03%)
Mutual labels: crawler, spider, web-crawler
Terpene Profile Parser For Cannabis Strains
Parser and database to index the terpene profile of different strains of Cannabis from online databases
Stars: ✭ 63 (-48.36%)
Mutual labels: crawler, scrapy, web-crawler
Crawler Detect
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Stars: ✭ 1,549 (+1169.67%)
Mutual labels: crawler, spider
Crawlab Lite
中文 | English
Lite version of Crawlab, golang-based web crawler management platform, supporting crawlers in any language.
Compared with Crawlab, this lite version focuses on crawler management on a single machine, it runs independent of any external database and removes a lot of non-essential features.
⚠️ This version is still in early development and some features may be unstable.
Quick Start
Docker Compose
- Create
docker-compose.yml
in any directory as follows:
version: '3'
services:
master:
image: zkqiang/crawlab-lite:latest
container_name: master
ports:
- "8080:8080"
volumes:
- "./data:/app/data" # persistent volume
- Run the command in this directory:
docker-compose up -d
- Visit
http://localhost:8080
Source Code
- Clone repository
git clone https://github.com/crawlab-team/crawlab-lite
cd crawlab-lite
- Run backend
cd backend
go run main.go
- Run frontend
cd ../frontend
npm i && npm run serve
- Visit
http://localhost:8080
Screenshot
Spider List
Task List
Schedule List
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].