jxltom / Scrapymon
Licence: mit
Simple Web UI for Scrapy spider management via Scrapyd
Stars: ✭ 35
Programming Languages
python
139335 projects - #7 most used programming language
Labels
Projects that are alternatives of or similar to Scrapymon
Python Spider
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
Stars: ✭ 615 (+1657.14%)
Mutual labels: scrapy
Funpyspidersearchengine
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
Stars: ✭ 782 (+2134.29%)
Mutual labels: scrapy
Scrapy Cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
Stars: ✭ 921 (+2531.43%)
Mutual labels: scrapy
Easy Scraping Tutorial
Simple but useful Python web scraping tutorial code.
Stars: ✭ 583 (+1565.71%)
Mutual labels: scrapy
Jspider
JSpider会每周更新至少一个网站的JS解密方式,欢迎 Star,交流微信:13298307816
Stars: ✭ 914 (+2511.43%)
Mutual labels: scrapy
House Renting
Possibly the best practice of Scrapy 🕷 and renting a house 🏡
Stars: ✭ 741 (+2017.14%)
Mutual labels: scrapy
Mailinglistscraper
A python web scraper for public email lists.
Stars: ✭ 19 (-45.71%)
Mutual labels: scrapy
Tweetscraper
TweetScraper is a simple crawler/spider for Twitter Search without using API
Stars: ✭ 694 (+1882.86%)
Mutual labels: scrapy
Scrapy Finance
[OUTDATED] scrapy spiders to crawl the financial text data 📚 📜 pertinent to train word vectors 🚀
Stars: ✭ 17 (-51.43%)
Mutual labels: scrapy
Icrawler
A multi-thread crawler framework with many builtin image crawlers provided.
Stars: ✭ 629 (+1697.14%)
Mutual labels: scrapy
Voyages Sncf Api
A scrapy spider that scraps times and prices from Voyages Sncf. It uses scrapyrt to provide an API interface.
Stars: ✭ 7 (-80%)
Mutual labels: scrapy
Place2live
Analysis of the characteristics of different countries
Stars: ✭ 30 (-14.29%)
Mutual labels: scrapy
Scrapy Azuresearch Crawler Samples
Scrapy as a Web Crawler for Azure Search Samples
Stars: ✭ 20 (-42.86%)
Mutual labels: scrapy
Pdf downloader
A Scrapy Spider for downloading PDF files from a webpage.
Stars: ✭ 18 (-48.57%)
Mutual labels: scrapy
scrapymon
Simple management UI for scrapyd. The demo is available at http://scrapymon.demo.jxltom.me/ with admin
for both username and password. Note that the demo will reset every 40 minutes and it may take some time to spin up if no one has accessed it for a while.
Features
- Show all projects from a Scrapyd server
- Show all versions of each project
- Show all spiders in each project
- Show all pending, running and finished jobs from a Scrapyd server
- Show logs of each job
- Schedule spiders run
- Cancel pending or running jobs
- Delete project or a specific version
- Http basic access authentication supported
- Served by Gevent for production use
Screenshots
Getting Started
-
Install by
pip install scrapymon
. -
Run by
scrapymon [--host=<host>] [--port=<port>] [--server=<address_with_port>] [--auth=<username:password>]
.- Default
--host
is0.0.0.0
- Default
--port
is5000
- Default
--server
ishttp://127.0.0.1:6800
- Default
--auth
isadmin:admin
- Default
-
Or you can run by
scrapymon
with valid environment variables$HOST
,$PORT
,$SCRAPYD_SERVER
and$BASIC_AUTH
.
TODO
- Support schedule a spider run with arguments.
- Highlighted and searcharable logs with catagories
- Logs auto refresh and pagination
- Create project via
addversion.json
. - Time Localization
- Add Dockerfile
Contributing
Contributions are welcomed!
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].