All Projects → justeat → Justsaying

justeat / Justsaying

Licence: apache-2.0
A light-weight message bus on top of AWS services (SNS and SQS).

Projects that are alternatives of or similar to Justsaying

Lambdaguard
AWS Serverless Security
Stars: ✭ 300 (+91.08%)
Mutual labels:  aws, sns, sqs
Aws Sdk Perl
A community AWS SDK for Perl Programmers
Stars: ✭ 153 (-2.55%)
Mutual labels:  aws, sns, sqs
Dazn Lambda Powertools
Powertools (logger, HTTP client, AWS clients, middlewares, patterns) for Lambda functions.
Stars: ✭ 501 (+219.11%)
Mutual labels:  aws, sns, sqs
Brighter
Command Dispatcher, Processor, and Distributed Task Queue
Stars: ✭ 1,393 (+787.26%)
Mutual labels:  sns, sqs, dotnet-core
Serverless
This is intended to be a repo containing all of the official AWS Serverless architecture patterns built with CDK for developers to use. All patterns come in Typescript and Python with the exported CloudFormation also included.
Stars: ✭ 1,048 (+567.52%)
Mutual labels:  aws, sns, sqs
Laravel Plain Sqs
Custom SQS connector for Laravel (or Lumen) that supports third-party, plain JSON messages
Stars: ✭ 91 (-42.04%)
Mutual labels:  aws, sqs
Aws Cli Cheatsheet
☁️ AWS CLI + JQ = Make life easier
Stars: ✭ 94 (-40.13%)
Mutual labels:  aws, sns
Foundatio
Pluggable foundation blocks for building distributed apps.
Stars: ✭ 1,365 (+769.43%)
Mutual labels:  aws, message-bus
Slimmessagebus
Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.
Stars: ✭ 120 (-23.57%)
Mutual labels:  message-bus, dot-net
Terraform Aws Rds Cloudwatch Sns Alarms
Terraform module that configures important RDS alerts using CloudWatch and sends them to an SNS topic
Stars: ✭ 56 (-64.33%)
Mutual labels:  aws, sns
Loafer
Asynchronous message dispatcher - Currently using asyncio and amazon SQS
Stars: ✭ 104 (-33.76%)
Mutual labels:  sns, sqs
Simpleue
PHP queue worker and consumer - Ready for AWS SQS, Redis, Beanstalkd and others.
Stars: ✭ 124 (-21.02%)
Mutual labels:  aws, sqs
Skater .net Obfuscator
Skater .NET Obfuscator is an obfuscation tool for .NET code protection. It implements all known software protection techniques and obfuscation algorithms.
Stars: ✭ 64 (-59.24%)
Mutual labels:  dotnet-core, dot-net
Lambda Native
Make .NET AWS Lambda functions start 10x faster using LambdaNative.
Stars: ✭ 64 (-59.24%)
Mutual labels:  aws, dotnet-core
Fluentmediator
🔀 FluentMediator is an unobtrusive library that allows developers to build custom pipelines for Commands, Queries and Events.
Stars: ✭ 128 (-18.47%)
Mutual labels:  message-bus, dotnet-core
Nsubstitute
A friendly substitute for .NET mocking libraries.
Stars: ✭ 1,646 (+948.41%)
Mutual labels:  dotnet-core, dot-net
Sqs Producer
Simple scaffolding for applications that produce SQS messages
Stars: ✭ 125 (-20.38%)
Mutual labels:  aws, sqs
Components Contrib
Community driven, reusable components for distributed apps
Stars: ✭ 131 (-16.56%)
Mutual labels:  sns, sqs
Sqs Consumer
Build Amazon Simple Queue Service (SQS) based applications without the boilerplate
Stars: ✭ 1,019 (+549.04%)
Mutual labels:  aws, sqs
Netbarcode
Barcode generation library written in C# and .NET Standard 2
Stars: ✭ 149 (-5.1%)
Mutual labels:  dotnet-core, dot-net

JustSaying

NuGet Gitter

Build status codecov

A helpful library for publishing and consuming events / messages over SNS (SNS / SQS as a message bus).

Getting started

Before you can start publishing or consuming messages, you want to configure the AWS client factory.

        CreateMeABus.DefaultClientFactory = () => new DefaultAwsClientFactory(new BasicAWSCredentials("accessKey", "secretKey"))

You will also need an ILoggerFactory (see here); if you do not want logging then you can use an empty logger factory like so:

    ILoggerFactory loggerFactory = NullLoggerFactory.Instance;

Publishing messages

Here's how to get up & running with simple message publishing.

1. Create a message object (POCO)

  • These can be as complex as you like (provided it is under 256k serialised as Json).
  • They must be derived from the abstract Message class (currently).
        public class OrderAccepted : Message
        {
            public OrderAccepted(int orderId)
            {
                OrderId = orderId;
            }
            public int OrderId { get; private set; }
        }

2. Registering publishers

  • You will need to tell JustSaying which messages you intend to publish so it can setup any missing topics for you.
  • In this case, we are telling it to publish the OrderAccepted messages.
  • The topic will be the message type.
          var publisher = CreateMeABus.WithLogging(loggerFactory)
                .InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSnsMessagePublisher<OrderAccepted>();

2.(a) Configuring publishing options

  • You can also specify some publishing options (such as how to handle failures) using a configuration object like so:
         CreateMeABus.WithLogging(loggerFactory)
                .InRegion(RegionEndpoint.EUWest1.SystemName)
                .ConfigurePublisherWith(c => { c.PublishFailureReAttempts = 3; c.PublishFailureBackoffMilliseconds = 50; })
                .WithSnsMessagePublisher<OrderAccepted>();

3. Publish a message

  • This can be done wherever you want within your application.
  • Simply pass the publisher object through using your IOC container.
  • In this case, we are publishing the fact that a given order has been accepted.
        publisher.Publish(new OrderAccepted(123456));

BOOM! You're done publishing!

Consuming messages

Here's how to get up & running with message consumption. We currently support SQS subscriptions only, but keep checking back for other methods too (HTTP, Kinesis) (although we are kinda at the mercy of AWS here for internal HTTP delivery...)

1. Create Handlers

  • We tell the stack to handle messages by implementing an interface which tells the handler our message type
  • Here, we're creating a handler for OrderAccepted messages.
  • This is where you pass on to your BLL layer.
  • We also need to tell the stack whether we handled the message as expected. We can say things got messy either by returning false, or bubbling up exceptions.
        public class OrderNotifier : IHandler<OrderAccepted>
        {
            public bool Handle(OrderAccepted message)
            {
                // Some logic here ...
                // e.g. bll.NotifyRestaurantAboutOrder(message.OrderId);
                return true;
            }
        }

2. Register a subscription

  • This can be done at the same time as your publications are set up.
  • There is no limit to the number of handlers you can add to a subscription.
  • You can specify message retention policies etc in your subscription for resiliency purposes.
  • In this case, we are telling JustSaying to keep 'OrderAccepted' messages for the default time, which is one minute. They will be thrown away if not handled in this time.
  • We are telling it to keep 'OrderFailed' messages for 1 minute and not to handle them again on failure for 30 seconds. These are the default values.
            CreateMeABus.WithLogging(loggerFactory)
                .InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSqsTopicSubscriber()
                .IntoQueue("CustomerOrders")
                .WithMessageHandler<OrderAccepted>(new OrderNotifier())
                .StartListening();

That's it. By calling StartListening() we are telling the stack to begin polling SQS for incoming messages.

2.(a) Subscription Configuration

  • In this case, we are telling JustSaying to keep 'OrderAccepted' messages for the default time, which is one minute. They will be thrown away if not handled in this time.
  • We are telling it to keep 'OrderFailed' messages for 5 mins, and not to handle them again on failure for 60 seconds
            CreateMeABus.WithLogging(loggerFactory)
                .InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSqsTopicSubscriber()
                .IntoQueue("CustomerOrders")
                    .ConfigureSubscriptionWith(c => { c.MessageRetentionSeconds = 60; })
                        .WithMessageHandler<OrderAccepted>(new NotifyCustomerOfAcceptedOrder())
                    .ConfigureSubscriptionWith(c => { c.MessageRetentionSeconds = 300; c.VisibilityTimeoutSeconds = 60; })
                        .WithMessageHandler<OrderFailed>(new NotifyCustomerOfFailedOrder())
                .StartListening();

2.(b) Configure Throttling

JustSaying throttles message handlers, which means JustSaying will limit the maximum number of messages being processed concurrently. The default limit is 8 threads per processor core, i.e. Environment.ProcessorCount * 8. We feel that this is a sensible number, but it can be overridden. This is useful for web apps with TCP thread restrictions. To override throttling you need to specify optional parameter when setting SqsTopicSubcriber

            .ConfigureSubscriptionWith(c => { c.MaxAllowedMessagesInFlight = 100; })
                .WithMessageHandler<OrderAccepted>(new NotifyCustomerOfAcceptedOrder())

2.(c) Control Handlers' life cycle

You can tell JustSaying to delegate the creation of your handlers to an IoC container. All you need to do is to implement IHandlerResolver interface and pass it along when registering your handlers.

CreateMeABus.WithLogging(loggerFactory)
            .InRegion(RegionEndpoint.EUWest1.SystemName)
            .WithSqsTopicSubscriber()
            .IntoQueue("CustomerOrders")
            .WithMessageHandler<OrderAccepted>(new HandlerResolver())

Interrogation

JustSaying provides you access to the Subscribers and Publishers message types via IAmJustInterrogating interface on the message bus.

            IAmJustSayingFluently bus = CreateMeABus.WithLogging(loggerFactory)
                .InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSnsMessagePublisher<OrderAccepted>();

            IInterrogationResponse response =((IAmJustInterrogating)bus).WhatDoIHave();

Logging

JustSaying stack will throw out the following named logs from NLog:

  • "JustSaying" * Information on the setup & your configuration (Info level). This includes all subscriptions, tenants, publication registrations etc. * Information on the number of messages handled & heartbeat of queue polling (Trace level). You can use this to confirm you're receiving messages. Beware, it can get big!
  • "EventLog" * A full log of all the messages you publish (including the Json serialised version). *

Here's a snippet of the expected configuration:

    <logger name="EventLog" minlevel="Trace" writeTo="logger-specfic-log" final="true" />
    <logger name="JustSaying" minlevel="Trace" writeTo="logger-specfic-log" final="true" />

      <target
         name="logger-specfic-log"
         xsi:type="File"
         fileName="${logdir}\${loggerspecificlogfilename}"
         layout="${standardlayout}"
         archiveFileName="${logdir}\${loggerspecificlogfilename}"
         archiveEvery="Hour"
         maxArchiveFiles="8784"
         concurrentWrites="true"
         keepFileOpen="false"
      />

Dead letter Queue (Error queue)

JustSaying supports error queues and this option is enabled by default. When a handler is unable to handle a message, JustSaying will attempt to re-deliver the message up to 5 times (handler retry count is configurable) and if the handler is still unable to handle the message then the message will be moved to an error queue. You can opt out during subscription configuration.

IAM Requirements

JustSaying requires the following IAM actions to run smoothly;

// SNS
sns:CreateTopic
sns:ListTopics
sns:SetSubscriptionAttributes
sns:Subscribe

// SQS
sqs:ChangeMessageVisibility
sqs:CreateQueue
sqs:DeleteMessage
sqs:GetQueueAttributes
sqs:GetQueueUrl
sqs:ListQueues
sqs:ReceiveMessage
sqs:SetQueueAttributes

An example policy would look like;

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues",
                "sns:ListTopics",
                "sns:SetSubscriptionAttributes"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ChangeMessageVisibility",
                "sqs:CreateQueue",
                "sqs:DeleteMessage",
                "sqs:GetQueueUrl",
                "sqs:GetQueueAttributes",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SetQueueAttributes"
            ],
            "Resource": "arn:aws:sqs:aws-region:aws-account-id:uk-myfeature-orderaccepted"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sns:CreateTopic",
                "sns:Publish",
                "sns:Subscribe"
            ],
            "Resource": "arn:aws:sqs:aws-region:aws-account-id:uk-orderaccepted"
        }
    ]
}

Power tool

JustSaying comes with a power tool console app that helps you manage your SQS queues from the command line. At this point, the power tool is only able to move an arbitrary number of messages from one queue to another.

JustSaying.Tools.exe move -from "source_queue_name" -to "destination_queue_name" -in "region" -count "1"

Contributing...

Please read the contributing guide.

The End.....

...Happy Messaging!...

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].