blehnen / Dotnetworkqueue
Projects that are alternatives of or similar to Dotnetworkqueue
DotNetWorkQueue
A producer / distributed consumer library for dot net applications. Dot net 4.6.1, 4.7.2, 4.8 and Dot net standard 2.0 are supported
High level features
- Queue / De-queue POCO for distributed processing
- Queue / Process LINQ statements; compiled or dyanamic (expressed as string)
- Re-occurring job scheduler
See the Wiki for more indepth documention
Installation
Base
- NuGet package DotNetWorkQueue
Transports
- NuGet package DotNetWorkQueue.Transport.Redis
- NuGet package DotNetWorkQueue.Transport.SqlServer
- NuGet package DotNetWorkQueue.Transport.SQLite
- NuGet package DotNetWorkQueue.Transport.PostgreSQL
Metrics
- NuGet package DotNetWorkQueue.AppMetrics
Differences between versions
Dot net standard 2.0 is missing the following features from the full framework versions
- No support for aborting threads when stopping the consumer queues
- No support for dynamic linq statements
Usage - POCO
[Producer - Sql server]
[Producer - SQLite]
[Producer - Redis]
https://github.com/blehnen/DotNetWorkQueue/blob/master/Source/Samples/Redis/RedisProducer/Program.cs
[Producer - PostgreSQL]
[Consumer - Sql server]
[Consumer - SQLite]
[Consumer - Redis]
https://github.com/blehnen/DotNetWorkQueue/blob/master/Source/Samples/Redis/RedisConsumer/Program.cs
[Consumer - PostgreSQL]
Usage - Linq Expression
You can choose to send Linq expressions to be executed instead. This has some advantages, as the producers and consumers are generic; they no longer need to be message specific. The below examples are not transport specifc and assume that any queue creation steps have already been performed.
NOTE: It's possbile for a producer to queue up work that a consumer cannot process. In order for a consumer to execute the Linq statement, all types must be resolvable. For dynamic statements, it's also possible to queue up work that doesn't compile due to syntax errors. That won't be discovered until the consumer dequeues the work.
####Example#####
[Producer] NOTE - if passing in the message or worker notifications as arguments to dynamic linq, you must cast them. The internal compiler treats those as objects. You can see this syntax in the examples below. That's not nessasry if using standard Linq expressions.
Message
(IReceivedMessage<MessageExpression>)
WorkerNotification
(IWorkerNotification)
If you are passing value types, you will need to parse them. Here is an example. The Guid and the int are both inside string literials and parsed via the built in dot.net methods.
var id = Guid.NewGuid();
var runTime = 200;
$"(message, workerNotification) => StandardTesting.Run(new Guid(\"{id}\"), int.Parse(\"{runTime}\"))"
This will produce a linq expression that can be compiled and executed by the consumer, assuming that it can resolve all of the types.
[Consumer] The consumer is generic; it can process any linq expression. However, it must be able to resolve all types that the linq expression uses. You may need to wire up an assembly resolver if your DLL's cannot be located.
https://msdn.microsoft.com/en-us/library/system.appdomain.assemblyresolve(v=vs.110).aspx
The above queue will process all Linq statements sent to the specified connection / queue.
[Considerations]
No sandboxing or checking for risky commands is performed. For instance, the below statement will cause your consumer host to exit.
"(message, workerNotification) => Environment.Exit(0)"
If you decide to allow configuration files to define dyanmic Linq statements (or if you cannot trust the producer), you should consider running the consumer in an application domain sandbox. Otherwise, the only thing stopping a command like the following from executing would be O/S user permissions.
"(message, workerNotification) => System.IO.Directory.Delete(@"C:\Windows\, true)"
Usage - Job Scheduler
Jobs may be scheduled using Schyntax format. The scheduler and consumers are seperate; schedulers don't process any work, they queue it for processing by a consumer. The standard LINQ consumers are used to process work enqueued by a scheduler / schedule.
Any LINQ statement that a linq producer supports can be scheduled using the scheduler.
Multiple schedulers with the same schedule may be ran if needed for redundancy. However, it's important that the clocks on the machines are in sync, or that the same time provider is injected into the schedulers and consumers. See the WIKI for more information on this.
Generally speaking, you may get funny results if you are using multiple machines and the clocks are not in sync. The server based transports tend to provide solutions for this if you can't sync the clocks of the local machines; see the WIKI.
See Schyntax for event scheduling format.
[Scheduler]
The scheduler and container must be kept in scope until you are done scheduling work or shutting down. No work will be queued if the scheduler is disposed or falls out of scope.
To consume / process scheduled jobs, a Linq Consumer is used
More examples
Building the source
You'll need VS2019 (any version) and you'll also need to install the dot net core 2.0 SDK
All references are either in NuGet or the \lib folder - building from Visual studio should restore any needed files.
License
Copyright � 2015-2020 Brian Lehnen
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 2.1 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with this program. If not, see http://www.gnu.org/licenses/.
3rd party Libraries
This library uses multiple 3rd party libaries, listed below.
[DotNetWorkQueue]
[DotNetWorkQueue.Transport.Redis]
[DotNetWorkQueue.Transport.SqlServer]
- None
[DotNetWorkQueue.Transport.SQLite]
[DotNetWorkQueue.Transport.SQLite.Microsoft]
[DotNetWorkQueue.Transport.PostgreSQL]
[DotNetWorkQueue.AppMetrics]
[Unit / Integration Tests]
Developed with:
