s3parcp 
s3parcp is a CLI wrapper around AWS's Go SDK's Downloader. This downloader provides a chunked parallel download implementation from s3 offering speeds faster than s3cp. The API is inspired by cp.
Installation
Linux
Debian (Ubuntu/Mint)
Download and install the .deb:
RELEASES=chanzuckerberg/s3parcp/releases
VERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)
DOWNLOAD=s3parcp_${VERSION}_linux_amd64.deb
curl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} -o s3parcp.deb
sudo dpkg -i s3parcp.deb
rm s3parcp.debFedora (RHEL/CentOS)
Download and install the .rpm:
RELEASES=chanzuckerberg/s3parcp/releases
VERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)
DOWNLOAD=s3parcp_${VERSION}_linux_amd64.rpm
curl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} -o s3parcp.rpm
sudo rpm -i s3parcp.rpm
rm s3parcp.rpmMacOS
Install via homebrew:
brew tap chanzuckerberg/tap
brew install s3parcpBinary
Download the appropriate binary for your platform:
RELEASES=chanzuckerberg/s3parcp/releases
PLATFORM=#linux,darwin,windows
VERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)
DOWNLOAD=s3parcp_${VERSION}_${PLATFORM}_amd64.tar.gz
curl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} | tar zxWindows
Download the appropriate binary for your platform:
RELEASES=chanzuckerberg/s3parcp/releases
VERSION=$(curl https://api.github.com/repos/${RELEASES}/latest | jq -r .name | sed s/^v//)
DOWNLOAD=s3parcp_${VERSION}_windows_amd64.tar.gz
curl -L https://github.com/${RELEASES}/download/v${VERSION}/${DOWNLOAD} | tar zxUsage
Usage:
s3parcp [OPTIONS] [Source] [Destination]
Application Options:
-p, --part-size= Part size in bytes of parts to be downloaded
-c, --concurrency= Download concurrency
-b, --buffer-size= Size of download buffer in bytes
--checksum Compare checksum if downloading or place checksum
in metadata if uploading
-r, --recursive Copy directories or folders recursively
--version Print the current version
--s3_url= A custom s3 API url (also available as an environment
variable 'S3PARCP_S3_URL', the flag takes precedence)
--max-retries= Max per chunk retries (default: 3)
--disable-ssl Disable SSL
--disable-cached-credentials Disable caching AWS credentials
-v, --verbose verbose logging
Help Options:
-h, --help Show this help message
Arguments:
Source: Source to copy from
Destination: Destination to copy to (Optional, defaults to source's base
name)
Examples
Uploading
s3parcp my/local/file s3://my-bucket/my-objectDownloading
s3parcp s3://my-bucket/my-object my/local/fileTuning Chunk Parameters
Note: These example parameters don't necessarily represent good parameters for your system. s3parcp uses sane defaults so it is recommended to use the default parameters unless you have reason to believe your values will work better.
PART_SIZE=1048576 # 1 MB
BUFFER_SIZE=10485760 # 10 MB
CONCURRENCY=8
s3parcp \
--part-size $PART_SIZE \
--concurrency $CONCURRENCY \
--buffer-size $BUFFER_SIZE \
my/local/file s3://my-bucket/my-objectUsing CRC32C Checksum
You must upload your file to s3 with s3parcp and the --checksum flag to use this feature for downloads.
Upload your file:
s3parcp --checksum my/local/file s3://my-bucket/my-objectThe checksum should be stored in the s3 object's metadata with the key x-amz-meta-crc32c-checksum.
Download your file:
s3parcp --checksum s3://my-bucket/my-object my/new/local/fileFeatures
checksum
This tool comes with a parallelized crc32c checksum validator. The AWS SDK does not support checksums for multipart downloads. If you include the --checksum flag when uploading a checksum of your file will be computed and stored in the object's metadata in s3 with the key x-amz-meta-crc32c-checksum. When downloading, the --checksum flag will compute an independent crc32c checksum of the downloaded file and compare it of the checksum in the object's metadata.