All Projects → zarusz → Slimmessagebus

zarusz / Slimmessagebus

Licence: apache-2.0
Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

Projects that are alternatives of or similar to Slimmessagebus

Foundatio
Pluggable foundation blocks for building distributed apps.
Stars: ✭ 1,365 (+1037.5%)
Mutual labels:  azure, redis, messaging, message-bus
Azure Event Hubs For Kafka
Azure Event Hubs for Apache Kafka Ecosystems
Stars: ✭ 124 (+3.33%)
Mutual labels:  azure, apache-kafka, kafka, messaging
Enqueue Dev
Message Queue, Job Queue, Broadcasting, WebSockets packages for PHP, Symfony, Laravel, Magento. DEVELOPMENT REPOSITORY - provided by Forma-Pro
Stars: ✭ 1,977 (+1547.5%)
Mutual labels:  kafka, redis, message-bus
Lightbus
RPC & event framework for Python 3
Stars: ✭ 149 (+24.17%)
Mutual labels:  bus, redis, messaging
Kafkactl
Command Line Tool for managing Apache Kafka
Stars: ✭ 177 (+47.5%)
Mutual labels:  apache-kafka, kafka, avro
Kafka Ui
Open-Source Web GUI for Apache Kafka Management
Stars: ✭ 230 (+91.67%)
Mutual labels:  apache-kafka, kafka, pub-sub
Kop
Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar
Stars: ✭ 159 (+32.5%)
Mutual labels:  apache-kafka, kafka, pub-sub
Kafka Storm Starter
Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
Stars: ✭ 728 (+506.67%)
Mutual labels:  apache-kafka, kafka, avro
psr-container-messenger
Message bus and queue for Mezzio with Symfony Messenger + Enqueue
Stars: ✭ 24 (-80%)
Mutual labels:  messaging, bus, message-bus
godsend
A simple and eloquent workflow for streaming messages to micro-services.
Stars: ✭ 15 (-87.5%)
Mutual labels:  messaging, bus, message-bus
Servicebus
Simple service bus for sending events between processes using amqp.
Stars: ✭ 415 (+245.83%)
Mutual labels:  bus, messaging, message-bus
Awesome Pulsar
A curated list of Pulsar tools, integrations and resources.
Stars: ✭ 57 (-52.5%)
Mutual labels:  apache-kafka, messaging, pub-sub
Open Bank Mark
A bank simulation application using mainly Clojure, which can be used to end-to-end test and show some graphs.
Stars: ✭ 81 (-32.5%)
Mutual labels:  kafka, avro
Community
一个仿照牛客网实现的讨论社区,不仅实现了基本的注册,登录,发帖,评论,点赞,回复功能,同时使用前缀树实现敏感词过滤,使用wkhtmltopdf生成长图和pdf,实现网站UV和DAU统计,并将用户头像等信息存于七牛云服务器。
Stars: ✭ 80 (-33.33%)
Mutual labels:  kafka, redis
Kaufmann ex
Kafka backed service library.
Stars: ✭ 86 (-28.33%)
Mutual labels:  kafka, avro
Nats Server
High-Performance server for NATS.io, the cloud and edge native messaging system.
Stars: ✭ 10,223 (+8419.17%)
Mutual labels:  messaging, message-bus
Karafka
Framework for Apache Kafka based Ruby and Rails applications development.
Stars: ✭ 1,223 (+919.17%)
Mutual labels:  apache-kafka, kafka
Dataengineeringproject
Example end to end data engineering project.
Stars: ✭ 82 (-31.67%)
Mutual labels:  kafka, redis
Rebus
🚌 Simple and lean service bus implementation for .NET
Stars: ✭ 1,733 (+1344.17%)
Mutual labels:  messaging, message-bus
Kukulcan
A REPL for Apache Kafka
Stars: ✭ 103 (-14.17%)
Mutual labels:  apache-kafka, kafka

SlimMessageBus

SlimMessageBus is a client façade for message brokers for .NET. It comes with implementations for specific brokers (Apache Kafka, Azure EventHub, MQTT/Mosquitto, Redis Pub/Sub) and also for in memory message passing (in-process communication). SlimMessageBus additionally provides request-response implementation over message queues.

Gitter GitHub license

Branch Build Status
master Build status
develop Build status
other Build status

Key elements of SlimMessageBus

  • Consumers:
    • IConsumer<in TMessage> - subscriber in pub/sub (or queue consumer)
    • IRequestHandler<in TRequest, TResponse> - request handler in request-response
  • Producers:
    • IPublishBus - publisher in pub/sub (or queue producer)
    • IRequestResponseBus - sender in req/resp
    • IMessageBus - extends IPublishBus and IRequestResponseBus
  • Misc:
    • IRequestMessage<TResponse> - marker for request messages
    • MessageBus - static accessor for current context IMessageBus

Docs

Packages

Name Descripton NuGet
SlimMessageBus The core API for SlimMessageBus NuGet
Transport providers
SlimMessageBus.Host.Kafka Transport provider for Apache Kafka NuGet
SlimMessageBus.Host.AzureServiceBus Transport provider for Azure Service Bus NuGet
SlimMessageBus.Host.AzureEventHub Transport provider for Azure Event Hubs NuGet
SlimMessageBus.Host.Redis Transport provider for Redis NuGet
SlimMessageBus.Host.Memory Transport provider implementation for in-process (in memory) message passing (no messaging infrastructure required) NuGet
SlimMessageBus.Host.Hybrid Bus implementation that composes the bus out of other transport providers and performs message routing NuGet
Serialization
SlimMessageBus.Host.Serialization.Json Serialization plugin for JSON (Json.NET library) NuGet
SlimMessageBus.Host.Serialization.Avro Serialization plugin for Avro (Apache.Avro library) NuGet
SlimMessageBus.Host.Serialization.Hybrid Plugin that delegates serialization to other serializers based on message type NuGet
Container
SlimMessageBus.Host.AspNetCore Integration for ASP.NET Core (DI adapter, config helpers) NuGet
SlimMessageBus.Host.Autofac DI adapter for Autofac container NuGet
SlimMessageBus.Host.Unity DI adapter for Unity container NuGet
SlimMessageBus.Host.ServiceLocator DI adapter for CommonServiceLocator NuGet

Typically your application components (business logic, domain) only need to depend on SlimMessageBus which is the facade, and ultimately your application hosting layer (ASP.NET, Windows Service, Console App) will reference and configure the other packages (SlimMessageBus.Host.*) which are the providers and plugins.

Samples

Check out the Samples folder.

Quick example

Some service (or domain layer) sends a message:

IMessageBus bus; // injected

await bus.Publish(new SomeMessage())

Another service (or application layer) handles the message:

public class SomeMessageConsumer : IConsumer<SomeMessage>
{
   public Task OnHandle(SomeMessage message, string name) // name = topic or queue name
   {
       // handle the message
   }
}

Note: It is also possible to avoid having to implement the interface IConsumer<T> (see here).

The bus also supports request-response implemented via queues (or topics - depending what the chosen transport provider supports). The sender side sends a request message:

var messageResponse = await bus.Send(new MessageRequest());

Note: It is possible to configure the bus to timeout a request when the response does not arrive within alloted time (see here).

The receiving side handles the request and replies back:

public class MessageRequestHandler : IRequestHandler<MessageRequest, MessageResponse>
{
   public async Task<MessageResponse> OnHandle(MessageRequest request, string name)
   {
      // handle the request message and return response
   }
}

The bus will ask the chosen DI to provide the consumer instances (SomeMessageConsumer, MessageRequestHandler).

The configuration somewhere in your service:

var builder = MessageBusBuilder.Create()
	
   // Pub/Sub example:
   .Produce<SomeMessage>(x => x.DefaultTopic("some-topic"))
   .Consume<SomeMessage>(x => x
      .Topic("some-topic")
      .WithConsumer<SomeMessageConsumer>()
      //.Group("some-kafka-consumer-group") //  Kafka provider specific
      //.SubscriptionName("some-azure-sb-topic-subscription") // Azure ServiceBus provider specific
   )
	
   // Use JSON for message serialization                
   .WithSerializer(new JsonMessageSerializer())
   // Use DI from ASP.NET Core (or Autofac, Unity, ServiceLocator)
   .WithDependencyResolver(new AspNetCoreMessageBusDependencyResolver(serviceProvider))
	
   // Use Apache Kafka transport provider
   .WithProviderKafka(new KafkaMessageBusSettings("localhost:9092"));
   // Use Azure Service Bus transport provider
   //.WithProviderServiceBus(...)
   // Use Azure Azure Event Hub transport provider
   //.WithProviderEventHub(...)
   // Use Redis transport provider
   //.WithProviderRedis(...)
   // Use in-memory transport provider
   //.WithProviderMemory(...)

// Build the bus from the builder. Message consumers will start consuming messages from the configured topics/queues of the chosen provider.
IMessageBus bus = builder.Build();

// Register bus in your DI

Basic in-process pub/sub messaging (for domain events)

This example shows how SlimMessageBus and SlimMessageBus.Host.Memory can be used to implement Domain Events pattern. The provider passes messages in the same app domain process (no external message broker is required).

The domain event is a simple POCO:

// domain event
public class OrderSubmittedEvent
{
   public Order Order { get; }
   public DateTime Timestamp { get; }

   public OrderSubmittedEvent(Order order) { ... }
}

The event handler implements the IConsumer<T> interface:

// domain event handler
public class OrderSubmittedHandler : IConsumer<OrderSubmittedEvent>
{
   public Task OnHandle(OrderSubmittedEvent e, string name)
   {
      Console.WriteLine("Customer {0} {1} just placed an order for:", e.Order.Customer.Firstname, e.Order.Customer.Lastname);
      foreach (var orderLine in e.Order.Lines)
      {
         Console.WriteLine("- {0}x {1}", orderLine.Quantity, orderLine.ProductId);
      }

      Console.WriteLine("Generating a shipping order...");
      return Task.Delay(1000);
   }
}

The domain handler (well, really the consumer) is obtained from dependency resolver at the time of event publication. It can be scoped (per web request, per unit of work) as configured in your favorite DI container.

Somewhere in your domain layer the domain event gets raised:

// aggregate root
public class Order
{
   public Customer Customer { get; }
   private IList<OrderLine> _lines = new List<OrderLine>();
   public OrderState State { get; private set; }

   public IEnumerable<OrderLine> Lines => _lines.AsEnumerable();

   public Order(Customer customer)
   {
      State = OrderState.New;
      Customer = customer;
   }

   public OrderLine Add(string productId, int quantity) { }

   public void Submit()
   {
      State = OrderState.Submitted;

      var e = new OrderSubmittedEvent(this);
      MessageBus.Current.Publish(e).Wait(); // raise domain event
   }
}

Some sample logic executed in your domain:

var john = new Customer("John", "Whick");

var order = new Order(john);
order.Add("id_machine_gun", 2);
order.Add("id_grenade", 4);

order.Submit(); // events fired here

Notice the static MessageBus.Current property might actually be configured to resolve a scoped IMessageBus instance (web request scoped).

The SlimMessageBus configuration for in-memory provider looks like this:

// Define the recipie how to create our IMessageBus
var mbb = MessageBusBuilder.Create()
   .Produce<OrderSubmittedEvent>(x => x.DefaultTopic(x.MessageType.Name))
   .Consume<OrderSubmittedEvent>(x => x.Topic(x.MessageType.Name).WithConsumer<OrderSubmittedHandler>())
   .WithDependencyResolver(new AutofacMessageBusDependencyResolver())
   .WithProviderMemory(new MemoryMessageBusSettings
   {
      // supress serialization and pass the same event instance to subscribers (events contain domain objects we do not want serialized, also we gain abit on speed)
      EnableMessageSerialization = false
   });

// Create the IMessageBus instance from the builder
IMessageBus bus = mbb.Build();

// Set the provider to resolve our bus - this setup will work as a singleton.
MessageBus.SetProvider(() => bus);

See the complete sample for ASP.NET Core where the handler and bus is web-request scoped.

Request-response over Kafka topics

Use case:

  • Some front-end web app needs to display downsized image (thumbnails) of large images to speed up page load.
  • The thumbnails are requested in the WebApi and are generated on demand (and cached to disk) by the Worker (unless they exist already).
  • WebApi and Worker exchange messages via Apache Kafka
  • Worker can be scaled out (more instances, more kafka partitions)

Front-end web app makes a call to resize an image DSC0862.jpg to 120x80 resolution, by using this URL:

https://localhost:56788/api/image/DSC3781.jpg/r/?w=120&h=80&mode=1

This gets handled by the WebApi method of the ImageController

private readonly IRequestResponseBus _bus;
// ...
[Route("{fileId}")]
public async Task<HttpResponseMessage> GetImageThumbnail(string fileId, ThumbnailMode mode, int w, int h)
{
   var thumbFileContent = // ... try to load content for the desired thumbnail w/h/mode/fileId
   if (thumbFileContent == null)
   {
      // Task will await until response comes back (or timeout happens). The HTTP request will be queued and IIS processing thread released.
      var thumbGenResponse = await _bus.Send(new GenerateThumbnailRequest(fileId, mode, w, h));
      thumbFileContent = await _fileStore.GetFile(thumbGenResponse.FileId);
   }
   return ServeStream(thumbFileContent);
}

The GenerateThumbnailRequest request is handled by a handler in one of the pool of Worker console apps.

public class GenerateThumbnailRequestHandler : IRequestHandler<GenerateThumbnailRequest, GenerateThumbnailResponse>
{
   public Task<GenerateThumbnailResponse> OnHandle(GenerateThumbnailRequest request, string name)
   {
      // some processing
      return new GenerateThumbnailResponse { FileId = thumbnailFileId };
   }
}

The response gets replied onto the originating WebApi instance and the Task resolves causing the queued HTTP request to serve the resized image thumbnail.

var thumbGenResponse = await _bus.Send(new GenerateThumbnailRequest(fileId, mode, w, h));

The message bus configuration for the WebApi:

private IMessageBus BuildMessageBus()
{
   // unique id across instances of this application (e.g. 1, 2, 3)
   var instanceId = Configuration["InstanceId"];
   var kafkaBrokers = Configuration["Kafka:Brokers"];

   var instanceGroup = $"webapi-{instanceId}";
   var instanceReplyTo = $"webapi-{instanceId}-response";

   var mbb = MessageBusBuilder.Create()
      .Produce<GenerateThumbnailRequest>(x =>
      {
         //x.DefaultTimeout(TimeSpan.FromSeconds(10)); // Default response timeout for this request type
         x.DefaultTopic("thumbnail-generation"); // Use this topic as default when topic is not specified in IMessageBus.Publish() for that message type
      })
      .ExpectRequestResponses(x =>
      {
         x.ReplyToTopic(instanceReplyTo); // Expect all responses to my reqests replied to this topic
         x.Group(instanceGroup); // Kafka consumer group	    
         x.DefaultTimeout(TimeSpan.FromSeconds(30)); // Default global response timeout
      })
      .WithDependencyResolver(new AutofacMessageBusDependencyResolver())
      .WithSerializer(new JsonMessageSerializer())
      .WithProviderKafka(new KafkaMessageBusSettings(kafkaBrokers));

   var messageBus = mbb.Build();
   return messageBus;
}

The message bus configuration for the Worker:

private static IMessageBus BuildMessageBus()
{
   // unique id across instances of this application (e.g. 1, 2, 3)
   var instanceId = Configuration["InstanceId"];
   var kafkaBrokers = Configuration["Kafka:Brokers"];

   var instanceGroup = $"worker-{instanceId}";
   var sharedGroup = "workers";

   var mbb = MessageBusBuilder.Create()
      .Handle<GenerateThumbnailRequest, GenerateThumbnailResponse>(s =>
      {
         s.Topic("thumbnail-generation", t =>
         {
            t.WithHandler<GenerateThumbnailRequestHandler>()
               .Group(sharedGroup) // kafka consumer group
               .Instances(3);
         });
      })
      .WithDependencyResolver(new AutofacMessageBusDependencyResolver())
      .WithSerializer(new JsonMessageSerializer())
      .WithProviderKafka(new KafkaMessageBusSettings(kafkaBrokers));

   var messageBus = mbb.Build();
   return messageBus;
}

Because topics are partitioned in Kafka, requests originating from WebApi instances will be distributed across all Worker instances. However, to fine tune this, message key providers should configured (see Kafka provider wiki and samples).

Check out the complete sample for image resizing.

Features

  • Types of messaging patterns supported:
    • Publish-subscribe
    • Request-response
    • Queues
    • Hybrid of the above (e.g. Kafka with multiple topic consumers in one group)
  • Modern async/await syntax and TPL
  • Fluent configuration
  • Because SlimMessageBus is a facade, you have the ability to swap broker implementations
    • Using nuget pull another broker provider
    • Reconfigure SlimMessageBus and retest your app
    • Try out the messaging middleware that works best for your app (Kafka vs. Redis) without having to rewrite your app.

Principles

  • The core of SlimMessageBus is "slim"
    • Simple, common and friendly API to work with messaging systems
    • No external dependencies.
    • The core interface can be used in domain model (e.g. DomainEvents)
  • Plugin architecture:
    • DI integration (Microsoft.Extensions.DependencyInjection, Autofac, CommonServiceLocator, Unity)
    • Message serialization (JSON, XML)
    • Use your favorite messaging broker as provider by simply pulling a nuget package
  • No threads created (pure TPL)
  • Async/Await support
  • Fluent configuration
  • Logging is done via Microsoft.Extensions.Logging.Abstractions, so that you can connect your favorite logger provider.

License

Apache License 2.0

Build

cd src
dotnet build
dotnet pack --output ../dist

NuGet packaged end up in dist folder

Testing

To run tests you need to update the respective appsettings.json to match your own cloud infrstructure or local infrastructure. SMB has some message brokers setup on Azure for integration tests (secrets not shared).

Run all tests:

dotnet test

Run all tests except integration tests which require local/cloud infrastructure:

dotnet test --filter Category!=Integration

Credits

Thanks to the following service cloud providers for providing free instances for our integration tests:

Other test instances are hosted in Azure and paid by the project maintainer. If you want to help and sponsor, please write to me.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].