All Projects → meowgorithm → pipedream

meowgorithm / pipedream

Licence: MIT license
Easy multipart uploads for Amazon S3, DigitalOcean Spaces and S3-compatible services

Programming Languages

go
31211 projects - #10 most used programming language
Makefile
30231 projects

Projects that are alternatives of or similar to pipedream

laravel-uppy-s3-multipart-upload
Multipart Uploads using Laravel, AWS S3, and Uppy
Stars: ✭ 30 (+76.47%)
Mutual labels:  s3, multipart-uploads
lambda-multipart-parser
This nodejs module will parse the multipart-form containing files and fields from the AWS lambda event object. It works very well parsing binary and text files.
Stars: ✭ 45 (+164.71%)
Mutual labels:  s3
backbeat
Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.
Stars: ✭ 51 (+200%)
Mutual labels:  s3
ses-email-client
Simple, serverless client for AWS SES. With this, you can send/read emails received by SES into S3 without purchasing AWS Workmail. If you only use SES for email marketing, you can also see and preview your SES templates in the browser
Stars: ✭ 21 (+23.53%)
Mutual labels:  s3
s3-fuzzer
🔐 A concurrent, command-line AWS S3 Fuzzer. Written in Go.
Stars: ✭ 43 (+152.94%)
Mutual labels:  s3
live-streaming-on-aws-with-mediastore
Live streaming on AWS with Amazon S3 automatically configures AWS Elemental MediaLive, Amazon S3 and Amazon CloudFront to ingest, encode, package and deliver a single source live stream through the AWS Cloud. The Solution provides 3 Encoding profiles to support 1080p through 288p HTTP live streaming (HLS) outputs.
Stars: ✭ 84 (+394.12%)
Mutual labels:  s3
lamba-thumbnailer
AWS S3 Video Thumbnailer with Lambda
Stars: ✭ 21 (+23.53%)
Mutual labels:  s3
go-drive
A simple cloud drive mapping web app supports local, FTP/SFTP, S3, OneDrive, WebDAV, Google Drive.
Stars: ✭ 184 (+982.35%)
Mutual labels:  s3
cubefs
CubeFS is a cloud native distributed storage platform.
Stars: ✭ 3,062 (+17911.76%)
Mutual labels:  s3
aws-certified-developer-associate-udemy-notes
AWS Certified Developer Associate Udemy Notes
Stars: ✭ 20 (+17.65%)
Mutual labels:  s3
react-deploy-s3
Deploy create react app's in AWS S3
Stars: ✭ 67 (+294.12%)
Mutual labels:  s3
data-transfer-hub
Seamless User Interface for replicating data into AWS.
Stars: ✭ 102 (+500%)
Mutual labels:  s3
terraform-aws-cloudtrail-s3-bucket
S3 bucket with built in IAM policy to allow CloudTrail logs
Stars: ✭ 38 (+123.53%)
Mutual labels:  s3
s3-credentials
A tool for creating credentials for accessing S3 buckets
Stars: ✭ 138 (+711.76%)
Mutual labels:  s3
terraform-aws-lb-s3-bucket
Terraform module to provision an S3 bucket with built in IAM policy to allow AWS Load Balancers to ship access logs
Stars: ✭ 29 (+70.59%)
Mutual labels:  s3
skbn
Copy files and directories between Kubernetes and cloud storage
Stars: ✭ 68 (+300%)
Mutual labels:  s3
saisoku
Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.
Stars: ✭ 40 (+135.29%)
Mutual labels:  s3
direct-upload-s3-signaturev4
Example of Directly uploading files to S3 with PHP
Stars: ✭ 50 (+194.12%)
Mutual labels:  s3
ob bulkstash
Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.
Stars: ✭ 113 (+564.71%)
Mutual labels:  s3
prefect-docker-compose
A simple guide to understand Prefect and make it work with your own docker-compose configuration.
Stars: ✭ 122 (+617.65%)
Mutual labels:  s3

Pipe Dream

Latest Release GoDoc Build Status

Easy multipart uploads for Amazon S3, DigitalOcean Spaces and S3-compatible services. Available as a CLI and Go library.

CLI

Usage

# Set your secrets, region and endpoint in the environment
export ACCESS_KEY="..."
export SECRET_KEY="..."
export ENDPOINT="sfo2.digitaloceanspaces.com" # for AWS set REGION

# Pipe in data or redirect in a file
pipedream --bucket images --path pets/puppy.jpg < puppy.jpg

# Get fancy
export now=$(date +"%Y-%m-%d_%H:%M:%S_%Z")
cat /data/dump.rdb | gzip | pipedream --bucket backups --path dump-$now.rdb.gz

# For more info
pipedream -h

Installation

Download a build from the releases page. macOS, Linux and Windows builds are available for a variety of architectures.

macOS users can also use Homebrew:

brew tap meowgorithm/tap && brew install meowgorithm/tap/pipedream

Or you can just use go get:

go get github.com/meowgorithm/pipedream/pipedream

Library

The library uses an event based model, sending events through a channel.

import "github.com/meowgorithm/pipedream"

// Create a new multipart upload object
m := pipedream.MultipartUpload{
    AccessKey: os.Getenv("ACCESS_KEY"),
    SecretKey: os.Getenv("SECRET_KEY"),
    Endpoint:  "sfo2.digitaloceanspaces.com", // you could use Region for AWS
    Bucket:    "my-fave-bucket",
}

// Get an io.Reader, like an *os.File or os.Stdout
f, err := os.Open("big-redis-dump.rdb")
if err != nil {
    fmt.Printf("Rats: %v\n", err)
    os.Exit(1)
}
defer f.Close()

// Send up the data. Pipdream returns a channel where you can listen for events
ch := m.Send(f, "backups/dump.rdb")
done := make(chan struct{})

// Listen for activity. For more detailed reporting, see the docs
go func() {
    for {
        e := <-ch
        switch e.(type) {
        case pipedream.Complete:
            fmt.Println("It worked!")
            close(done)
            return
        case pipedream.Error:
            fmt.Println("Rats, it didn't work.")
            close(done)
            return
        }
    }
}()

<-done

Full source of this example. For an example with more detailed reporting, see the source code in the CLI.

Awknowledgements

Thanks to to Apoorva Manjunath‘s S3 multipart upload example for the S3 implementation details.

License

MIT

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].