All Projects → bgahagan → Git Remote S3

bgahagan / Git Remote S3

Licence: mit
Push and pull git repos to/from an s3 bucket, encrypted using gpg.

Programming Languages

rust
11053 projects

Projects that are alternatives of or similar to Git Remote S3

docker base images
Vlad's Base Images for Docker
Stars: ✭ 61 (-60.39%)
Mutual labels:  s3, gpg
Git Plus
Git utilities
Stars: ✭ 150 (-2.6%)
Mutual labels:  command-line-tool
Pueue
🌠 Manage your shell commands.
Stars: ✭ 2,471 (+1504.55%)
Mutual labels:  command-line-tool
Gopass Ui
gopass CLI + UI = visual cross-platform password manager for teams
Stars: ✭ 147 (-4.55%)
Mutual labels:  gpg
Artisan Menu
📝 Artisan Menu - Use Artisan via an elegant console GUI
Stars: ✭ 141 (-8.44%)
Mutual labels:  command-line-tool
Json 2 Kotlin
Convert JSON to Kotlin Data Classes
Stars: ✭ 148 (-3.9%)
Mutual labels:  command-line-tool
Wal G
Archival and Restoration for Postgres
Stars: ✭ 1,974 (+1181.82%)
Mutual labels:  s3
Kafka Connect Storage Cloud
Kafka Connect suite of connectors for Cloud storage (Amazon S3)
Stars: ✭ 153 (-0.65%)
Mutual labels:  s3
Caesium Clt
Caesium Command Line Tools - Lossy/lossless image compression tool using mozjpeg and zopflipng
Stars: ✭ 149 (-3.25%)
Mutual labels:  command-line-tool
Onhold
🔊 Play sounds while and after shell jobs complete
Stars: ✭ 146 (-5.19%)
Mutual labels:  command-line-tool
Mc
MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.
Stars: ✭ 1,962 (+1174.03%)
Mutual labels:  s3
Hubot Grafana
📈🤖 Query Grafana dashboards
Stars: ✭ 141 (-8.44%)
Mutual labels:  s3
Pins
Pin, Discover and Share Resources
Stars: ✭ 149 (-3.25%)
Mutual labels:  s3
Scar
Deploy static websites in seconds - with HTTPS, a global CDN, and custom domains.
Stars: ✭ 1,715 (+1013.64%)
Mutual labels:  s3
Dejavu
Quickly detect already witnessed data.
Stars: ✭ 151 (-1.95%)
Mutual labels:  command-line-tool
Hledger
A reliable, user-friendly Plain Text Accounting tool with command line, terminal and web interfaces.
Stars: ✭ 1,887 (+1125.32%)
Mutual labels:  command-line-tool
Georaptor
Python Geohash Compression Tool
Stars: ✭ 143 (-7.14%)
Mutual labels:  command-line-tool
Xioc
Extract indicators of compromise from text, including "escaped" ones.
Stars: ✭ 148 (-3.9%)
Mutual labels:  command-line-tool
Aws Sdk Perl
A community AWS SDK for Perl Programmers
Stars: ✭ 153 (-0.65%)
Mutual labels:  s3
0x4447 product s3 email
📫 A serverless email server on AWS using S3 and SES
Stars: ✭ 2,905 (+1786.36%)
Mutual labels:  s3

Git Remote S3 Helper

Push and pull git repos to/from an s3 bucket. Uses gpg to encrypt the repo contents (but not branch names!) before sending to s3.

This likely most useful for small teams who don't want to host their own private repository, but still want to manage their own encryption. For example, my use case is periodically backing up a repo from a desktop and pull to a laptop to develop remotely.

Example Usage

Add a remote using the s3 transport:

git remote add s3remote s3://my_bucket/prefix

And then you can push/pull to the remote as usual:

git pull s3remote master

git push s3remote

Or even clone from s3:

git clone s3://my_bucket/prefix

Installation

  • Put git-remote-s3 in your PATH
    • Download the latest release here, gunzip and put it in your PATH
    • Or, install using cargo: cargo install git-remote-s3
  • Make sure s3 credentials are setup
    • See here for details on how the rusoto library loads as credentials (similar to the aws command line).
  • Setup gpg
    • gpg encryption will be attempted using git config user.email as a recipient. You'll want to ensure you have public and private keys setup for this user.
    • Alternatively, you can set a list of space-delimited recipients using the remote.<name>.gpgRecipientsconfig.

Design Notes

Due to the eventual consistency behaviour of s3, the semantics of pushing are slightly different when pushing to a 'proper' git repository. An attempt is made to prevent non-force pushes that do not include the current head as an ancestor (as proper git repos do), but eventual consistency means this is not guaranteed. Its possible for multiple heads to exist for the same branch, in which case the clients consider the newest head to be the truth. All heads for a branch can be seen using git ls-remote - the latest (newest) head the have the branch's name; older head will be shown using the naming scheme: <branch_name>__<sha>. An old head is retained until a new head is pushed that includes the old head as an ancestor, at which point the old head is deleted. This prevents any data loss, but puts the burden on the user to manually merge in old branches.

Each branch is stored (after being bundled with git bundle and encrypted with gpg) on s3 using the key s3://bucket/prefix/<ref_name>/<sha>.bundle. On average, a git push will incur two list, a put and a delete s3 operation. A git pull will incur a list and a get s3 operation.

Future improvements

  • A better way to notify the user there are multiple heads on s3.
    • Show warning when attempting to push/fetch and there are multiple heads for a branch?
  • Allow disabling gpg with remote.<name>.gpg
  • use gpg.program
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].