爬虫管理平台
主要功能
- 查看Scrapyd节点状态；
- 查看Scrapyd节点下存在的Project和Spider;
- 支持对Spider的分组功能；
- 对Spider组和单个Spider进行控制（运行、暂停、继续、停止）；
- 实时显示分组及Spiders的工作状态。
安装依赖
- 需要先安装 pipenv，通过 直接在程序目录执行
pipenv install。
ToonoW / SpiderManager
Licence: Apache-2.0 License
爬虫管理平台
Stars: ✭ 27
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to SpiderManager
ARGUS
ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9
Stars: ✭ 68 (+151.85%)
Mutual labels: scrapy, scrapyd
Spiderkeeper
admin ui for scrapy/open source scrapinghub
Stars: ✭ 2,562 (+9388.89%)
Mutual labels: scrapy, scrapyd
Scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO 👉
Stars: ✭ 2,385 (+8733.33%)
Mutual labels: scrapy, scrapyd
logparser
A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API of Scrapyd.
Stars: ✭ 70 (+159.26%)
Mutual labels: scrapy, scrapyd
Gerapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
Stars: ✭ 2,601 (+9533.33%)
Mutual labels: scrapy, scrapyd
hk0weather
Web scraper project to collect the useful Hong Kong weather data from HKO website
Stars: ✭ 49 (+81.48%)
Mutual labels: scrapy
python-spider
python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取（生成cookie）、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答
Stars: ✭ 45 (+66.67%)
Mutual labels: scrapy
proxi
Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.
Stars: ✭ 32 (+18.52%)
Mutual labels: scrapy
pythonSpider
🕷️some python spiders with BeautifulSoup or scarpy
Stars: ✭ 28 (+3.7%)
Mutual labels: scrapy
ImageGrabber
A Scrapy demo : Download all images from a site
Stars: ✭ 33 (+22.22%)
Mutual labels: scrapy
scrapy facebooker
Collection of scrapy spiders which can scrape posts, images, and so on from public Facebook Pages.
Stars: ✭ 22 (-18.52%)
Mutual labels: scrapy
Data-Engineering-Projects
Personal Data Engineering Projects
Stars: ✭ 167 (+518.52%)
Mutual labels: scrapy
restaurant-finder-featureReviews
Build a Flask web application to help users retrieve key restaurant information and feature-based reviews (generated by applying market-basket model – Apriori algorithm and NLP on user reviews).
Stars: ✭ 21 (-22.22%)
Mutual labels: scrapy
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].