All Projects → w-zengtao → rb-spider

w-zengtao / rb-spider

Licence: MIT license
基于 RabbitMQ 中间件的爬虫的 Ruby 实现 [Developing]

Programming Languages

ruby
36898 projects - #4 most used programming language
shell
77523 projects

Projects that are alternatives of or similar to rb-spider

scrapy-distributed
A series of distributed components for Scrapy. Including RabbitMQ-based components, Kafka-based components, and RedisBloom-based components for Scrapy.
Stars: ✭ 38 (+192.31%)
Mutual labels:  spider, rabbitmq
Jlitespider
A lite distributed Java spider framework :-)
Stars: ✭ 151 (+1061.54%)
Mutual labels:  spider, rabbitmq
aioamqp consumer
consumer/producer/rpc library built over aioamqp
Stars: ✭ 36 (+176.92%)
Mutual labels:  rabbitmq
alicemq
RabbitMQ Visualizer
Stars: ✭ 102 (+684.62%)
Mutual labels:  rabbitmq
python-spider
python爬虫小项目【持续更新】【笔趣阁小说下载、Tweet数据抓取、天气查询、网易云音乐逆向、天天基金网查询、微博数据抓取(生成cookie)、有道翻译逆向、企查查免登陆爬虫、大众点评svg加密破解、B站用户爬虫、拉钩免登录爬虫、自如租房字体加密、知乎问答
Stars: ✭ 45 (+246.15%)
Mutual labels:  spider
docker-symfony
Docker Symfony (PHP-FPM - NGINX - MySQL - MailHog - Redis - RabbitMQ)
Stars: ✭ 32 (+146.15%)
Mutual labels:  rabbitmq
jobSpider
jobSpider是一只scrapy爬虫,用于爬取职位信息
Stars: ✭ 28 (+115.38%)
Mutual labels:  spider
LearningPoint
A repository for learning different technologies, frameworks, features......
Stars: ✭ 66 (+407.69%)
Mutual labels:  rabbitmq
zucc xk ZhengFang
ZUCC正方教务系统抢课助手。针对ZUCC正方教务系统模拟登录,爬取课程信息,自动抓包发包抢课。具体实现流程可参考README中的实现原理链接
Stars: ✭ 40 (+207.69%)
Mutual labels:  spider
benchmark-http
No description or website provided.
Stars: ✭ 15 (+15.38%)
Mutual labels:  spider
ng4-stompjs-demo
A sample using Angular4, Angular CLI and @stom/ng2-stompjs
Stars: ✭ 20 (+53.85%)
Mutual labels:  rabbitmq
PHP-interview-myway
记录PHP面试包括计算机网路,操作系统,PHP,redis,数据库MySQL(从2019年12月开始)面试了好未来,跟谁学,头条,极客保险,百度文库,百度知道等等公司。持续更新中
Stars: ✭ 380 (+2823.08%)
Mutual labels:  rabbitmq
QQSpider
爬取QQ用户信息(qq号、昵称、生日、地址等基本信息)并做简要analysis。
Stars: ✭ 21 (+61.54%)
Mutual labels:  spider
MusicSpider
Music Spider. Go 👾 Music Spider 是使用Golang写的音乐聚合爬虫,目前支持的站点包括 网易、QQ、虾米、酷狗、百度。
Stars: ✭ 24 (+84.62%)
Mutual labels:  spider
docker-rabbitmq-node
🐳 A playground for Docker with RabbitMQ and Node.
Stars: ✭ 32 (+146.15%)
Mutual labels:  rabbitmq
nabbitmq
Node.js library for interacting with RabbitMQ based on RxJS streams
Stars: ✭ 20 (+53.85%)
Mutual labels:  rabbitmq
Spydan
A web spider for shodan.io without using the Developer API.
Stars: ✭ 30 (+130.77%)
Mutual labels:  spider
opentelemetry-ext-js
js extensions for the open-telemetry project
Stars: ✭ 122 (+838.46%)
Mutual labels:  rabbitmq
youdao
有道词典网页爬虫
Stars: ✭ 22 (+69.23%)
Mutual labels:  spider
docker-workshop-with-react-aspnetcore-redis-rabbitmq-mssql
An Asp.Net Core Docker workshop project that includes react ui, redis, mssql, rabbitmq and azure pipelines
Stars: ✭ 53 (+307.69%)
Mutual labels:  rabbitmq

Spider

Maintainability codebeat badge Build Status

Docs

Install

gem install spider -s https://github.com/w-zengtao/rb-spider

Development

The gem is available as open source under the terms of the MIT License.

What I am doing ...

  • Confused by how to design the Timer

    # it seems that this is not a good way
    
    def interval seconds
      loop do
        sleep seconds
      end
    end
    
    interval do
      puts 'the code'
    end
    # And then use a thread to polling redis seems good
  • Create the Task class to extend our mission

By every commit, I record my thinking about evolution of this gem

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].