GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → awslabs → data-transfer-hub

awslabs / data-transfer-hub

Licence: Apache-2.0 license
Seamless User Interface for replicating data into AWS.

Programming Languages

typescript
32286 projects
javascript
184084 projects - #8 most used programming language
SCSS
7915 projects
shell
77523 projects
Velocity Template Language
3 projects
HTML
75241 projects
python
139335 projects - #7 most used programming language

Labels

awscloudformationreplications3amazon-ecrdata-replicationamazon-s3

Projects that are alternatives of or similar to data-transfer-hub

Aws Data Replication Hub
Seamless User Interface for replicating data into AWS.
Stars: ✭ 40 (-60.78%)
Mutual labels:  cloudformation, replication, s3
Litestream
Streaming replication for SQLite.
Stars: ✭ 3,795 (+3620.59%)
Mutual labels:  replication, s3
Seaweedfs
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
Stars: ✭ 13,380 (+13017.65%)
Mutual labels:  replication, s3
backbeat
Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.
Stars: ✭ 51 (-50%)
Mutual labels:  replication, s3
awesome-storage
A curated list of storage open source tools. Backups, redundancy, sharing, distribution, encryption, etc.
Stars: ✭ 324 (+217.65%)
Mutual labels:  replication, s3
radio
Redundant Array of Distributed Independent Objectstores in short RADIO performs synchronous mirroring, erasure coding across multiple object stores
Stars: ✭ 25 (-75.49%)
Mutual labels:  replication, s3
Hands-On-Serverless-Applications-with-Go
Hands-On Serverless Applications with Go, published by Packt.
Stars: ✭ 92 (-9.8%)
Mutual labels:  cloudformation, s3
go-localstack
Go Wrapper for using localstack
Stars: ✭ 56 (-45.1%)
Mutual labels:  cloudformation, s3
Awesome Aws
A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.
Stars: ✭ 9,895 (+9600.98%)
Mutual labels:  cloudformation, s3
Sumologic Aws Lambda
A collection of lambda functions to collect data from Cloudwatch, Kinesis, VPC Flow logs, S3, security-hub and AWS Inspector
Stars: ✭ 126 (+23.53%)
Mutual labels:  cloudformation, s3
backup-repository
Backup storage for E2E GPG-encrypted files, with multi-user, quotas, versioning, using a object storage (S3/Min.io/GCS etc.) and deployed on Kubernetes or standalone.
Stars: ✭ 21 (-79.41%)
Mutual labels:  s3, amazon-s3
Aws Sdk Perl
A community AWS SDK for Perl Programmers
Stars: ✭ 153 (+50%)
Mutual labels:  cloudformation, s3
Minio
High Performance, Kubernetes Native Object Storage
Stars: ✭ 30,698 (+29996.08%)
Mutual labels:  s3, amazon-s3
Wal E
Continuous Archiving for Postgres
Stars: ✭ 3,313 (+3148.04%)
Mutual labels:  replication, s3
fluent-bit-go-s3
[Deprecated] The predessor of fluent-bit output plugin for Amazon S3. https://aws.amazon.com/s3/
Stars: ✭ 34 (-66.67%)
Mutual labels:  s3, amazon-s3
Aws Toolkit Vscode
AWS Toolkit for Visual Studio Code, an extension for working with AWS services including AWS Lambda.
Stars: ✭ 823 (+706.86%)
Mutual labels:  cloudformation, s3
Scar
Deploy static websites in seconds - with HTTPS, a global CDN, and custom domains.
Stars: ✭ 1,715 (+1581.37%)
Mutual labels:  cloudformation, s3
aws-pdf-textract-pipeline
🔍 Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
Stars: ✭ 141 (+38.24%)
Mutual labels:  cloudformation, s3
syncflux
SyncFlux is an Open Source InfluxDB Data synchronization and replication tool for migration purposes or HA clusters
Stars: ✭ 145 (+42.16%)
Mutual labels:  data-replication
pglogrepl
PostgreSQL logical replication library for Go.
Stars: ✭ 150 (+47.06%)
Mutual labels:  replication
View All Similar Projects ➔

Data Transfer Hub

Note: If you have already deployed this solution, refer to the User Guide.

Data Transfer Hub is a secure, reliable, scalable, and trackable solution offers a unified user experience that allows you to easily create and manage different types of data transfer tasks from different sources to Amazon Web Services cloud-native services. You can launch this solution and start to replicate data within a few minutes.

With Data Transfer Hub, you can perform any of the following tasks:

  • Transfer Amazon S3 objects between AWS China Regions and AWS Regions.
  • Transfer Amazon S3 objects between AWS GovCloud and AWS Regions.
  • Transfer data from other cloud providers’ object storage services (including Alibaba Cloud OSS, Tencent COS, Qiniu Kodo, and S3 Compatible storage) to Amazon S3.
  • Transfer Amazon ECR images between AWS China Regions and AWS Regions.
  • Transfer container images from public container registries (e.g., Docker Hub, Google gcr.io, Red Hat Quay.io) to Amazon ECR.

Features

  • Authentication (Cognito User Pool, OIDC)
  • Self-service User Interface
  • CDK Deployment
  • CloudFormation Deployment
  • S3 Plugin
    • Amazon S3 object copy between AWS China Regions and AWS Regions.
    • Amazon S3 object copy between AWS GovCloud and AWS Regions.
    • Data transfer from Alibaba Cloud OSS to Amazon S3
    • Data transfer from Tencent COS to Amazon S3
    • Data transfer from Qiniu Kodo to Amazon S3
    • Support data transfer with Metadata
    • Support S3 Events to trigger data transfer
    • Use TCP BBR to improve network performance.
    • Support transfer from Amazon S3 Compatible Storage
  • ECR Plugin
    • Amazon ECR images copy between AWS Regions and AWS China Regions
    • Public docker registry to AWS ECR images copy
    • Copy all images or only selected Images
    • Support One-time images copy
    • Support Incremental images copy

Architecture

A web portal is launched in the customer's AWS account. Through the web portal, customers can create data transfer tasks and manage them in a centralized place. When the user create a data transfer task through the web portal, the backend invokes the CloudFormation API to deploy another stack to provision all resources to start data transfer. The solution only provision resources when needed. You can also choose to deploy the data transfer plugin independently.

Available Plugins:

Solution Deployment

Time to deploy: Approximately 15 minutes.

Refer to this guide if you want deploy this solution in AWS China Regions.

Launch CloudFormation Stack

Follow the step-by-step instructions to configure and deploy the Data Transfer Hub into your account.

  1. Make sure you have sign in AWS Console already.

  2. Click the following button to launch the CloudFormation Stack in your account.

    Launch Stack

  3. Input AdminEmail parameter. An email containing the temporary password is sent to this email. You'll need access to this email to sign in.

  4. Click Next and select Create Stack.

Login into the Data Transfer Hub Portal

An email containing the temporary password will be sent to the provided email. Note the password for later use.

  1. Open the CloudFormation Console, and choose the deployed stack.
  2. Select the Output of the CloudFormation stack, and find the PortalUrl.
  3. Open the portal in your browser.
  4. Sign in with the AdminEmail and the temporary password.
  5. Set a new password.
  6. Verify your email address and turn on account recovery (optional).
  7. The page redirects to the login page.
  8. Input the AdminEmail and the new password.

User Manual

Create your first data transfer task, For the complete user guide, refer to User Guide for more information.

FAQ

Q. Which are the supported Reigons of this solution?
You can deploy this solution in these Reigons: N.Virginia (us-east-1), Ohio (us-east-2), N.California (us-west-1), Oregon (us-west-2), Mumbai (ap-south-1), Seoul (ap-northeast-2), Singapore (ap-southeast-1), Sydney (ap-southeast-2), Tokyo (ap-northeast-1), Canada (ca-central-1), Frankfurt (eu-central-1), Ireland (eu-west-1), London (eu-west-2), Paris (eu-west-3), Stockholm (eu-north-1), São Paulo (sa-east-1), Beijing (cn-north-1), Ningxia (cn-northwest-1).

Q. If I want to copy data from U.S. Regions to China Regions, where should I deploy the solution?
You can either deploy the solution in U.S. Regions or deploy the solution in China Regions.

Q. Can I deploy the solution in AWS Account A, and transfer Amazon S3 objects from Account B to Account C?
Yes. In this case, you will need to store the AccessKeyID and SecretAccessKey for Account B and C in the Secrets Manager of Account A.

More Resources

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].