fredwu / Opq
A simple, in-memory queue with worker pooling and rate limiting in Elixir.
Stars: ✭ 178
Programming Languages
elixir
2628 projects
Projects that are alternatives of or similar to Opq
Task Easy
A simple, customizable, and lightweight priority queue for promises.
Stars: ✭ 244 (+37.08%)
Mutual labels: in-memory, queue
k2hash
K2HASH - NoSQL Key Value Store(KVS) library
Stars: ✭ 33 (-81.46%)
Mutual labels: queue, in-memory
Honeydew
Job Queue for Elixir. Clustered or Local. Straight BEAM. Optional Ecto. 💪🍈
Stars: ✭ 670 (+276.4%)
Mutual labels: pool, queue
Laravel Queue
Laravel Enqueue message queue extension. Supports AMQP, Amazon SQS, Kafka, Google PubSub, Redis, STOMP, Gearman, Beanstalk and others
Stars: ✭ 155 (-12.92%)
Mutual labels: queue
Rainbowminer
GPU/CPU Mining script with intelligent profit-switching between miningpools, algorithms, miners, using all possible combinations of devices (NVIDIA, AMD, CPU). Features: actively maintained, uses the top actual miner programs (Bminer, Ccminer, Claymore, Dstm, EnemyZ, Sgminer, T-rex and more) easy setup wizard, webinterface, auto update.
Stars: ✭ 158 (-11.24%)
Mutual labels: pool
Unifrost
Making it easier to push pubsub events directly to the browser.
Stars: ✭ 166 (-6.74%)
Mutual labels: in-memory
Fastdep
Fast integration dependencies in spring boot.是一个快速集成依赖的框架,集成了一些常用公共的依赖。例:多数据源,Redis,JWT...
Stars: ✭ 178 (+0%)
Mutual labels: pool
Algo Tree
Algo-Tree is a collection of Algorithms and data structures which are fundamentals to efficient code and good software design. Creating and designing excellent algorithms is required for being an exemplary programmer. It contains solutions in various languages such as C++, Python and Java.
Stars: ✭ 166 (-6.74%)
Mutual labels: queue
Acl
Server framework and network components written by C/C++ for Linux, Mac, FreeBSD, Solaris(x86), Windows, Android, IOS
Stars: ✭ 2,113 (+1087.08%)
Mutual labels: queue
Phive Queue
$queue->push('I can be popped off after', '10 minutes');
Stars: ✭ 161 (-9.55%)
Mutual labels: queue
Node Rethinkdb Job Queue
A persistent job or task queue backed by RethinkDB.
Stars: ✭ 158 (-11.24%)
Mutual labels: queue
Redux Data Structures
Reducer factory functions for common data structures: counters, maps, lists (queues, stacks), sets, etc.
Stars: ✭ 157 (-11.8%)
Mutual labels: queue
Holster
A place to keep useful golang functions and small libraries
Stars: ✭ 166 (-6.74%)
Mutual labels: queue
Laravel Rate Limited Job Middleware
A job middleware to rate limit jobs
Stars: ✭ 166 (-6.74%)
Mutual labels: queue
OPQ: One Pooled Queue
A simple, in-memory queue with worker pooling and rate limiting in Elixir. OPQ leverages Erlang's queue module and Elixir's GenStage.
Originally built to support Crawler.
Features
- A fast, in-memory FIFO queue.
- Worker pool.
- Rate limit.
- Timeouts.
- Pause / resume / stop the queue.
Usage
A simple example:
{:ok, opq} = OPQ.init
OPQ.enqueue(opq, fn -> IO.inspect("hello") end)
OPQ.enqueue(opq, fn -> IO.inspect("world") end)
Specify module, function and arguments:
{:ok, opq} = OPQ.init
OPQ.enqueue(opq, IO, :inspect, ["hello"])
OPQ.enqueue(opq, IO, :inspect, ["world"])
Specify a custom name for the queue:
OPQ.init(name: :items)
OPQ.enqueue(:items, fn -> IO.inspect("hello") end)
OPQ.enqueue(:items, fn -> IO.inspect("world") end)
Specify a custom worker to process items in the queue:
defmodule CustomWorker do
def start_link(item) do
Task.start_link fn ->
Agent.update(:bucket, &[item | &1])
end
end
end
Agent.start_link(fn -> [] end, name: :bucket)
{:ok, opq} = OPQ.init(worker: CustomWorker)
OPQ.enqueue(opq, "hello")
OPQ.enqueue(opq, "world")
Agent.get(:bucket, & &1) # => ["world", "hello"]
Rate limit:
{:ok, opq} = OPQ.init(workers: 1, interval: 1000)
Task.async fn ->
OPQ.enqueue(opq, fn -> IO.inspect("hello") end)
OPQ.enqueue(opq, fn -> IO.inspect("world") end)
end
If no interval is supplied, the ratelimiter will be bypassed.
Check the queue and number of available workers:
{:ok, opq} = OPQ.init
OPQ.enqueue(opq, fn -> Process.sleep(1000) end)
{queue, available_workers} = OPQ.info(opq) # => {:normal, {[], []}, 9}
Process.sleep(1200)
{queue, available_workers} = OPQ.info(opq) # => {:normal, {[], []}, 10}
Stop the queue:
{:ok, opq} = OPQ.init
OPQ.enqueue(opq, fn -> IO.inspect("hello") end)
OPQ.stop(opq)
OPQ.enqueue(opq, fn -> IO.inspect("world") end) # => (EXIT) no process...
Pause and resume the queue:
{:ok, opq} = OPQ.init
OPQ.enqueue(opq, fn -> IO.inspect("hello") end) # => "hello"
OPQ.pause(opq)
OPQ.info(opq) # => {:paused, {[], []}, 10}
OPQ.enqueue(opq, fn -> IO.inspect("world") end)
OPQ.resume(opq) # => "world"
OPQ.info(opq) # => {:normal, {[], []}, 10}
Configurations
Option | Type | Default Value | Description |
---|---|---|---|
:name |
atom/module | pid | The name of the queue. |
:worker |
module | OPQ.Worker |
The worker that processes each item from the queue. |
:workers |
integer | 10 |
Maximum number of workers. |
:interval |
integer | 0 |
Rate limit control - number of milliseconds before asking for more items to process, defaults to 0 which is effectively no rate limit. |
:timeout |
integer | 5000 |
Number of milliseconds allowed to perform the work, it should always be set to higher than :interval . |
Changelog
Please see CHANGELOG.md.
License
Licensed under MIT.
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].