GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → jamesmcm → s3rename

jamesmcm / s3rename

Licence: Apache-2.0, MIT licenses found Licenses found Apache-2.0 LICENSE-APACHE MIT LICENSE-MIT
Tool to mass-rename S3 keys

Programming Languages

rust
11053 projects

Labels

s3hacktoberfest

Projects that are alternatives of or similar to s3rename

reaction-file-collections-sa-s3
An S3 storage adapter for Reaction Commerce's reaction-file-collections
Stars: ✭ 14 (-12.5%)
Mutual labels:  s3
cachegrand
cachegrand is an open-source fast, scalable and secure Key-Value store, also fully compatible with Redis protocol, designed from the ground up to take advantage of modern hardware vertical scalability, able to provide better performance and a larger cache at lower cost, without losing focus on distributed systems.
Stars: ✭ 87 (+443.75%)
Mutual labels:  s3
serverless-s3bucket-sync
Serverless Plugin to sync local folders with an S3 bucket
Stars: ✭ 24 (+50%)
Mutual labels:  s3
s3-monitoring
No description or website provided.
Stars: ✭ 14 (-12.5%)
Mutual labels:  s3
flask-drive
A simple Flask app to upload and download files off Amazon's S3
Stars: ✭ 23 (+43.75%)
Mutual labels:  s3
minback-postgres
A container which provides the ability to backup a PostgreSQL database to Minio on demand
Stars: ✭ 18 (+12.5%)
Mutual labels:  s3
s3-edit
Edit directly a file on Amazon S3 in CLI
Stars: ✭ 69 (+331.25%)
Mutual labels:  s3
terraform-aws-frontend
Collection of Terraform modules for frontend app deployment on AWS.
Stars: ✭ 31 (+93.75%)
Mutual labels:  s3
poto
multi cloud storage to image gallery + image proxy + file api - 350 LOC.
Stars: ✭ 20 (+25%)
Mutual labels:  s3
Bucket-Flaws
Bucket Flaws ( S3 Bucket Mass Scanner ): A Simple Lightweight Script to Check for Common S3 Bucket Misconfigurations
Stars: ✭ 43 (+168.75%)
Mutual labels:  s3
esop
Cloud-enabled backup and restore tool for Apache Cassandra
Stars: ✭ 40 (+150%)
Mutual labels:  s3
go-storage
A vendor-neutral storage library for Golang: Write once, run on every storage service.
Stars: ✭ 387 (+2318.75%)
Mutual labels:  s3
terraform-aws-s3-object
Terraform module which creates S3 object resources on AWS
Stars: ✭ 15 (-6.25%)
Mutual labels:  s3
terraform-aws-efs-backup
Terraform module designed to easily backup EFS filesystems to S3 using DataPipeline
Stars: ✭ 40 (+150%)
Mutual labels:  s3
terraform-remote-state
A Terraform module that configures an s3 bucket for use with Terraform's remote state feature
Stars: ✭ 21 (+31.25%)
Mutual labels:  s3
tidysq
tidy processing of biological sequences in R
Stars: ✭ 29 (+81.25%)
Mutual labels:  s3
great-migration
Copy objects from Rackspace to S3
Stars: ✭ 15 (-6.25%)
Mutual labels:  s3
crux.land
crux.land is a free registry service meant for hosting small (≤ 20kB) single deno scripts.
Stars: ✭ 50 (+212.5%)
Mutual labels:  s3
hyperblock
[DEPRECATED] A theoretically-infinitely-scalable Skyblock plugin.
Stars: ✭ 16 (+0%)
Mutual labels:  s3
aws-s3-bucket-purger
A program that will purge any AWS S3 Bucket of objects and versions quickly
Stars: ✭ 18 (+12.5%)
Mutual labels:  s3
View All Similar Projects ➔

s3rename

s3rename is a tool to mass-rename keys within an S3 bucket.

The interface is designed to mimic the Perl rename utility on GNU/Linux (also known as prename and perl-rename).

s3rename uses asynchronous requests to rename the keys in parallel, as fast as possible.

The expression provided is applied to the entire key, allowing you to rename parent "directories".

Object properties are preserved, unless the --no-preserve-properties flag is used.

Object ACL (Access Control List) settings will also be preserved, unless the --no-preserve-acl flag is used.

It is highly recommended to use the --dry-run flag at first to ensure the changes reflect what you intend.

Usage

Note that regardless of the prefix used for filtering in the S3 URL provided, the regex is applied to the whole key. This is necessary to allow for full changes of the directory structure.

USAGE:
    s3rename [FLAGS] [OPTIONS] <expr> <s3-url>

FLAGS:
    -n, --dry-run                   Do not carry out modifications (only print)
    -h, --help                      Prints help information
        --no-anonymous-groups       Do not allow anonymous capture groups i.e. \1, \2 - may be useful when dealing with
                                    keys containing backslashes
        --no-preserve-acl           Do not preserve Object ACL settings (all will be set to private)
        --no-preserve-properties    Do not preserve object properties (saves retrieving per-object details) - using this
                                    flag will remove any encryption
        --no-overwrite              Do not overwrite existing keys
    -q, --quiet                     Do not print key modifications
    -V, --version                   Prints version information
    -v, --verbose                   Print debug messages

OPTIONS:
        --aws-region <aws-region>    AWS Region (will be taken from bucket region if not overridden here)
        --canned-acl <canned-acl>    Canned access_control_list override - sets this ACL for all renamed keys [possible
                                     values: private, public-read, public-read-write, aws-exec-read, authenticated-read,
                                     bucket-owner-read, bucket-owner-full-control]

ARGS:
    <expr>      Perl RegEx Replace Expression (only s/target/replacement/flags form supported)
    <s3-url>    S3 URL: s3://bucket-name/optional-key-prefix

Examples

s3rename uses the Perl regular expression format (like sed) to rename files:

$ aws s3 ls s3://s3rename-test-bucket --recursive
2020-05-01 12:30:25         16 testnewfile.txt

$ ./s3rename "s/new/old" s3://s3rename-test-bucket/test
Renaming testnewfile.txt to testoldfile.txt

$ aws s3 ls s3://s3rename-test-bucket --recursive
2020-05-01 12:33:48         16 testoldfile.txt

The --dry-run flag will print changes to be made without carrying them out. This is highly recommended before running changes.

By default ACL settings for objects will be preserved (unless --no-preserve-acl is passed), however this does not apply to ACL settings which depend on the bucket ACL (i.e. public write access).

The --canned-acl <canned-acl> option can be used to set the ACL of all renamed objects to the provided canned ACL. Note that some canned ACLs are affected by bucket settings (such as public-read-write).

Renaming flat files to a nested directory structure for AWS Glue

This program was originally inspired by the need to rename the keys of thousands of files which were stored in a flat structure, so that they could be correctly parsed by AWS Glue which requires a nested structure with the "directory" names corresponding to the partitions.

$ aws s3 ls s3://s3rename-test-bucket/datatest --recursive
2020-05-01 12:38:33          0 datatest/
2020-05-01 12:38:43          0 datatest/data_2020-04-01.txt
2020-05-01 12:38:43          0 datatest/data_2020-04-02.txt
2020-05-01 12:38:43          0 datatest/data_2020-04-03.txt
2020-05-01 12:38:43          0 datatest/data_2020-04-04.txt
2020-05-01 12:38:43          0 datatest/data_2020-04-05.txt
2020-05-01 12:38:43          0 datatest/data_2020-05-01.txt
2020-05-01 12:38:43          0 datatest/data_2020-05-02.txt
2020-05-01 12:38:43          0 datatest/data_2020-06-01.txt

$ ./s3rename 's/data_(?P<year>[0-9]{4})-(?P<month>[0-9]{2})-(?P<day>[0-9]{2}).txt/year=$year\/month=$month\/day=$day\/data_$year-$month-$day.txt/g' s3://s3rename-test-bucket/datatest
Renaming datatest/ to datatest/
Renaming datatest/data_2020-04-01.txt to datatest/year=2020/month=04/day=01/data_2020-04-01.txt
Renaming datatest/data_2020-04-02.txt to datatest/year=2020/month=04/day=02/data_2020-04-02.txt
Renaming datatest/data_2020-04-03.txt to datatest/year=2020/month=04/day=03/data_2020-04-03.txt
Renaming datatest/data_2020-04-04.txt to datatest/year=2020/month=04/day=04/data_2020-04-04.txt
Renaming datatest/data_2020-04-05.txt to datatest/year=2020/month=04/day=05/data_2020-04-05.txt
Renaming datatest/data_2020-05-01.txt to datatest/year=2020/month=05/day=01/data_2020-05-01.txt
Renaming datatest/data_2020-05-02.txt to datatest/year=2020/month=05/day=02/data_2020-05-02.txt
Renaming datatest/data_2020-06-01.txt to datatest/year=2020/month=06/day=01/data_2020-06-01.txt

$ aws s3 ls s3://s3rename-test-bucket/datatest --recursive
2020-05-01 12:38:33          0 datatest/
2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=01/data_2020-04-01.txt
2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=02/data_2020-04-02.txt
2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=03/data_2020-04-03.txt
2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=04/data_2020-04-04.txt
2020-05-01 12:39:38          0 datatest/year=2020/month=04/day=05/data_2020-04-05.txt
2020-05-01 12:39:38          0 datatest/year=2020/month=05/day=01/data_2020-05-01.txt
2020-05-01 12:39:38          0 datatest/year=2020/month=05/day=02/data_2020-05-02.txt
2020-05-01 12:39:38          0 datatest/year=2020/month=06/day=01/data_2020-06-01.txt

Note the use of single quotes for the sed regex string to avoid issues with the $ symbols in the shell.

You can also use anonymous capture groups, with the replacement parts marked either by $ or , i.e.:

's/data_([0-9]{4})-([0-9]{2})-([0-9]{2}).txt/year=\1\/month=\2\/day=\3\/data_\1-\2-\3.txt/g'

is equivalent to the above, and equivalent to:

's/data_([0-9]{4})-([0-9]{2})-([0-9]{2}).txt/year=$1\/month=$2\/day=$3\/data_$1-$2-$3.txt/g'

Use multiple dollar symbols to escape the dollars (for literal dollar symbols).

Installation

s3rename depends on OpenSSL at runtime.

Building from source requires a Rust toolchain and Cargo.

If you use this tool please consider starring the Github repo and voting for the package on the AUR.

Arch Linux (via the Arch User Repository (AUR))

Using the yay AUR helper:

$ yay -S s3rename

Alternatively you can manually install the package from the AUR.

Cargo (via crates.io)

$ cargo install s3rename

The s3rename binary will then be in your Cargo binaries directory (and this should already be on your $PATH.

Cargo (from this repository)

s3rename can be installed via Cargo from this cloned repository:

$ git clone [email protected]:jamesmcm/s3rename.git
$ cd s3rename
$ cargo install --path .

The s3rename binary will then be in your Cargo binaries directory (and this should already be on your $PATH.

Linux x86_64 binary

Static binaries compiled for Linux x86_64 are available in the Github releases.

Known Issues

  • Buckets and objects using S3 Object Lock are currently unsupported.
  • Expiry rules set with prefixes in the bucket properties will not be updated (so any keys moved out of the scope of these rules will no longer have the expiry rules applied). In the future a specific command to update expiry rules may be added.
  • s3rename does not support custom encryption keys for encrypted buckets (i.e. if your encryption key is not generated and stored by AWS). This could be added in a future version.
  • The rename operation is not fully atomic (since it involves separate CopyObject and DeleteObject requests) - this means that if s3rename is terminated suddenly during operation, the bucket could be left with copied files where the originals have not been renamed (re-running s3rename with the same arguments would fix this).

S3 Billing

s3rename operates on keys within the same bucket and so should trigger no data transfer costs.

Whilst it does use CopyObjectRequests to carry out the renaming, the additional data does not exist for long and should trigger no costs for data usage:

Regarding billing for data storage, the S3 Billing documentation states:

The volume of storage billed in a month is based on the average storage used throughout the month. This includes all object data and metadata stored in buckets that you created under your AWS account. We measure your storage usage in “TimedStorage-ByteHrs,” which are added up at the end of the month to generate your monthly charges.

License

s3rename is licensed under either of:

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].