All Projects → jxltom → Scrapymon

jxltom / Scrapymon

Licence: mit
Simple Web UI for Scrapy spider management via Scrapyd

Programming Languages

python
139335 projects - #7 most used programming language

Labels

Projects that are alternatives of or similar to Scrapymon

Python Spider
豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章
Stars: ✭ 615 (+1657.14%)
Mutual labels:  scrapy
Funpyspidersearchengine
Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索
Stars: ✭ 782 (+2134.29%)
Mutual labels:  scrapy
Scrapy Cluster
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
Stars: ✭ 921 (+2531.43%)
Mutual labels:  scrapy
Scrapyrt
HTTP API for Scrapy spiders
Stars: ✭ 637 (+1720%)
Mutual labels:  scrapy
Jd spider
两只蠢萌京东的分布式爬虫.
Stars: ✭ 738 (+2008.57%)
Mutual labels:  scrapy
Seeker
Seeker - another job board aggregator.
Stars: ✭ 16 (-54.29%)
Mutual labels:  scrapy
Easy Scraping Tutorial
Simple but useful Python web scraping tutorial code.
Stars: ✭ 583 (+1565.71%)
Mutual labels:  scrapy
Jspider
JSpider会每周更新至少一个网站的JS解密方式,欢迎 Star,交流微信:13298307816
Stars: ✭ 914 (+2511.43%)
Mutual labels:  scrapy
House Renting
Possibly the best practice of Scrapy 🕷 and renting a house 🏡
Stars: ✭ 741 (+2017.14%)
Mutual labels:  scrapy
Mailinglistscraper
A python web scraper for public email lists.
Stars: ✭ 19 (-45.71%)
Mutual labels:  scrapy
Faster Than Requests
Faster requests on Python 3
Stars: ✭ 639 (+1725.71%)
Mutual labels:  scrapy
Tweetscraper
TweetScraper is a simple crawler/spider for Twitter Search without using API
Stars: ✭ 694 (+1882.86%)
Mutual labels:  scrapy
Scrapy Finance
[OUTDATED] scrapy spiders to crawl the financial text data 📚 📜 pertinent to train word vectors 🚀
Stars: ✭ 17 (-51.43%)
Mutual labels:  scrapy
Icrawler
A multi-thread crawler framework with many builtin image crawlers provided.
Stars: ✭ 629 (+1697.14%)
Mutual labels:  scrapy
Voyages Sncf Api
A scrapy spider that scraps times and prices from Voyages Sncf. It uses scrapyrt to provide an API interface.
Stars: ✭ 7 (-80%)
Mutual labels:  scrapy
Pythonspidernotes
Python入门网络爬虫之精华版
Stars: ✭ 5,634 (+15997.14%)
Mutual labels:  scrapy
Py3 scripts
Life is short, *****.
Stars: ✭ 5 (-85.71%)
Mutual labels:  scrapy
Place2live
Analysis of the characteristics of different countries
Stars: ✭ 30 (-14.29%)
Mutual labels:  scrapy
Scrapy Azuresearch Crawler Samples
Scrapy as a Web Crawler for Azure Search Samples
Stars: ✭ 20 (-42.86%)
Mutual labels:  scrapy
Pdf downloader
A Scrapy Spider for downloading PDF files from a webpage.
Stars: ✭ 18 (-48.57%)
Mutual labels:  scrapy

scrapymon

Simple management UI for scrapyd. The demo is available at http://scrapymon.demo.jxltom.me/ with admin for both username and password. Note that the demo will reset every 40 minutes and it may take some time to spin up if no one has accessed it for a while.

Features

  • Show all projects from a Scrapyd server
  • Show all versions of each project
  • Show all spiders in each project
  • Show all pending, running and finished jobs from a Scrapyd server
  • Show logs of each job
  • Schedule spiders run
  • Cancel pending or running jobs
  • Delete project or a specific version
  • Http basic access authentication supported
  • Served by Gevent for production use

Screenshots

projects_dash jobs_dash logs_dash

Getting Started

  • Install by pip install scrapymon.

  • Run by scrapymon [--host=<host>] [--port=<port>] [--server=<address_with_port>] [--auth=<username:password>].

    • Default --host is 0.0.0.0
    • Default --port is 5000
    • Default --server is http://127.0.0.1:6800
    • Default --auth is admin:admin
  • Or you can run by scrapymon with valid environment variables $HOST, $PORT, $SCRAPYD_SERVER and $BASIC_AUTH.

TODO

  • Support schedule a spider run with arguments.
  • Highlighted and searcharable logs with catagories
  • Logs auto refresh and pagination
  • Create project via addversion.json.
  • Time Localization
  • Add Dockerfile

Contributing

Contributions are welcomed!

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].