All Projects → fujiwara → Rin

fujiwara / Rin

Licence: MIT license
Rin is a Redshift data Importer by SQS messaging.

Programming Languages

go
31211 projects - #10 most used programming language
Makefile
30231 projects
Dockerfile
14818 projects

Projects that are alternatives of or similar to Rin

go-localstack
Go Wrapper for using localstack
Stars: ✭ 56 (+107.41%)
Mutual labels:  sqs, redshift
django-eb-sqs-worker
Django Background Tasks for Amazon Elastic Beanstalk
Stars: ✭ 27 (+0%)
Mutual labels:  sqs
Simpleue
PHP queue worker and consumer - Ready for AWS SQS, Redis, Beanstalkd and others.
Stars: ✭ 124 (+359.26%)
Mutual labels:  sqs
Alpine Sqs
Dockerized ElasticMQ server + web UI over Alpine Linux for local development
Stars: ✭ 183 (+577.78%)
Mutual labels:  sqs
Components Contrib
Community driven, reusable components for distributed apps
Stars: ✭ 131 (+385.19%)
Mutual labels:  sqs
aws-developer-associate-certificate
Note to pass the AWS Developer Associate Exam
Stars: ✭ 53 (+96.3%)
Mutual labels:  sqs
Loafer
Asynchronous message dispatcher - Currently using asyncio and amazon SQS
Stars: ✭ 104 (+285.19%)
Mutual labels:  sqs
toiler
Toiler is a AWS SQS long-polling thread-based message processor.
Stars: ✭ 15 (-44.44%)
Mutual labels:  sqs
ontopic
Display SNS messages on your terminal
Stars: ✭ 20 (-25.93%)
Mutual labels:  sqs
Kombu
Kombu is a messaging library for Python.
Stars: ✭ 2,263 (+8281.48%)
Mutual labels:  sqs
Laravel Bridge
Package to use Laravel on AWS Lambda with Bref
Stars: ✭ 168 (+522.22%)
Mutual labels:  sqs
Aws Sdk Perl
A community AWS SDK for Perl Programmers
Stars: ✭ 153 (+466.67%)
Mutual labels:  sqs
PyRSMQ
Python Implementation of Redis Simple Message Queue Algorithm
Stars: ✭ 35 (+29.63%)
Mutual labels:  sqs
Sqs Producer
Simple scaffolding for applications that produce SQS messages
Stars: ✭ 125 (+362.96%)
Mutual labels:  sqs
mq-go
SQS Consumer Server for Go
Stars: ✭ 28 (+3.7%)
Mutual labels:  sqs
Amazon Sqs Java Messaging Lib
This Amazon SQS Java Messaging Library holds the Java Message Service compatible classes, that are used for communicating with Amazon Simple Queue Service.
Stars: ✭ 117 (+333.33%)
Mutual labels:  sqs
Sqs Worker Serverless
Example for SQS Worker in AWS Lambda using Serverless
Stars: ✭ 164 (+507.41%)
Mutual labels:  sqs
amazon-sns-java-extended-client-lib
This AWS SNS client library allows to publish messages to SNS that exceed the 256 KB message size limit.
Stars: ✭ 23 (-14.81%)
Mutual labels:  sqs
blaster
Web hooks for message queues
Stars: ✭ 14 (-48.15%)
Mutual labels:  sqs
sqsiphon
No description or website provided.
Stars: ✭ 19 (-29.63%)
Mutual labels:  sqs

Rin

Rin is a Redshift data Importer by SQS messaging.

Architecture

  1. (Someone) creates a S3 object.
  2. S3 event notifications will send to a message to SQS.
  3. Rin will fetch messages from SQS, and publish a "COPY" query to Redshift.

Installation

Binary packages

Releases

Homebrew

$ brew install fujiwara/tap/rin

Docker

GitHub Packages

$ docker pull ghcr.io/fujiwara/rin:v1.1.3

Configuration

Configuring Amazon S3 Event Notifications.

  1. Create SQS queue.
  2. Attach SQS access policy to the queue. Example Walkthrough 1:
  3. Enable Event Notifications on a S3 bucket.
  4. Run rin process with configuration for using the SQS and S3.

config.yaml

queue_name: my_queue_name    # SQS queue name

credentials:
  aws_region: ap-northeast-1

redshift:
  host: localhost
  port: 5439
  dbname: test
  user: test_user
  password: '{{ must_env "REDSHIFT_PASSWORD" }}'
  schema: public
  reconnect_on_error: true # disconnect Redshift on error occurred

s3:
  bucket: test.bucket.test
  region: ap-northeast-1

sql_option: "JSON 'auto' GZIP"       # COPY SQL option

# define import target mappings
targets:
  - s3:
      key_prefix: test/foo/ignore
    discard: true  # Do not import and do not try following targets. Matches only.

- redshift:
      table: foo
    s3:
      key_prefix: test/foo

  - redshift:
      schema: xxx
      table: bar
    s3:
      key_prefix: test/bar
    break: true       # Do not try following targets.

  - redshift:
      schema: $1      # expand by key_regexp captured value.
      table: $2
    s3:
      key_regexp: test/schema-([a-z]+)/table-([a-z]+)/

  - redshift:
      host: redshift.example.com       # override default section in this target
      port: 5439
      dbname: example
      user: example_user
      password: example_pass
      schema: public
      table: example
    s3:
      bucket: redshift.example.com
      region: ap-northeast-1
      key_prefix: logs/example/
    sql_option: "CSV DELIMITER ',' ESCAPE"

A configuration file is parsed by kayac/go-config.

go-config expands environment variables using syntax {{ env "FOO" }} or {{ must_env "FOO" }} in a configuration file.

When the password for Redshift is empty, Rin will try call GetClusterCredentials API to get a temporary password for the cluster.

Credentials

Rin requires credentials for SQS and Redshift.

  1. credentials.aws_access_key_id and credentials.aws_secret_access_key
  • used for SQS and Redshift.
  1. credentials.aws_iam_role
  • used for Redshift only.
  • for SQS, Rin will try to get a instance credentials.

Run

daemon mode

Rin waits new SQS messages and processing it continually.

$ rin -config config.yaml [-debug]

-config also accepts HTTP/S3/File URL to specify the location of configuration file. For example,

$ rin -config s3://rin-config.my-bucket/config.yaml

batch mode

Rin process new SQS messages and exit.

$ rin -config config.yaml -batch [-debug]
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].