All Projects β†’ jasonwhite β†’ Rudolfs

jasonwhite / Rudolfs

Licence: mit
A high-performance, caching Git LFS server with an AWS S3 and local storage back-end.

Programming Languages

rust
11053 projects

Projects that are alternatives of or similar to Rudolfs

Lru Cache
πŸ’« A feature complete LRU cache implementation in C++
Stars: ✭ 162 (+32.79%)
Mutual labels:  cache, server
Javatech
β˜•οΈ 汇总 Java εΌ€ε‘δΈ­εΈΈθ§ηš„δΈ»ζ΅ζŠ€ζœ―ηš„εΊ”η”¨γ€η‰Ήζ€§γ€εŽŸη†γ€‚
Stars: ✭ 310 (+154.1%)
Mutual labels:  cache, server
punic
Punic is a remote cache CLI built for Carthage and Apple .xcframework
Stars: ✭ 25 (-79.51%)
Mutual labels:  cache, aws-s3
Rome
Carthage cache for S3, Minio, Ceph, Google Storage, Artifactory and many others
Stars: ✭ 724 (+493.44%)
Mutual labels:  aws-s3, cache
node-backend-template
A template for NodeJS backend projects
Stars: ✭ 19 (-84.43%)
Mutual labels:  cache, aws-s3
Deta cache
ηΌ“ε­˜cacheζœεŠ‘ε™¨
Stars: ✭ 106 (-13.11%)
Mutual labels:  cache, server
Mailinabox
Mail-in-a-Box helps individuals take back control of their email by defining a one-click, easy-to-deploy SMTP+everything else server: a mail server in a box.
Stars: ✭ 10,649 (+8628.69%)
Mutual labels:  server
Ddn
DDN, Data Delivery Network, a next generation blockchain system
Stars: ✭ 118 (-3.28%)
Mutual labels:  server
Rust Server
Provides a dedicated linux server for Rust (the game) running inside a Docker container.
Stars: ✭ 114 (-6.56%)
Mutual labels:  server
Quicssh
SSH over QUIC
Stars: ✭ 116 (-4.92%)
Mutual labels:  server
Openapi Core
OpenAPI core
Stars: ✭ 119 (-2.46%)
Mutual labels:  server
Ipokemon Server
iPokeMon Server.
Stars: ✭ 119 (-2.46%)
Mutual labels:  server
Vertx Mqtt
Vert.x MQTT
Stars: ✭ 117 (-4.1%)
Mutual labels:  server
Node Restify
The future of Node.js REST development
Stars: ✭ 10,335 (+8371.31%)
Mutual labels:  server
Craft Blitz
Intelligent static page caching for creating lightning-fast sites with Craft CMS.
Stars: ✭ 118 (-3.28%)
Mutual labels:  cache
Laravel Parse
A Parse SDK bridge for Laravel 5
Stars: ✭ 116 (-4.92%)
Mutual labels:  server
Lru Cache Node
A lighting fast cache manager for node with least-recently-used policy.
Stars: ✭ 120 (-1.64%)
Mutual labels:  cache
Xtcp
A TCP Server Framework with graceful shutdown, custom protocol.
Stars: ✭ 116 (-4.92%)
Mutual labels:  server
Netclient Ios
Versatile HTTP Networking in Swift
Stars: ✭ 117 (-4.1%)
Mutual labels:  cache
Gdk For Unity Fps Starter Project
SpatialOS GDK for Unity FPS Starter Project
Stars: ✭ 119 (-2.46%)
Mutual labels:  server

Rudolfs

Build Status Crates.io AUR version

A high-performance, caching Git LFS server with an AWS S3 back-end.

Features

  • Multiple backends:

    1. AWS S3 backend with an optional local disk cache.
    2. Local disk backend.
  • A configurable local disk cache to speed up downloads (and reduce your S3 bill).

  • Corruption-resilient local disk cache. Even if the disk is getting blasted by cosmic rays, it'll find corrupted LFS objects and purge them from the cache transparently. The client should never notice this happening.

  • Encryption of LFS objects in both the cache and in permanent storage.

  • Separation of GitHub organizations and projects. Just specify the org and project names in the URL and they are automatically created. If two projects share many LFS objects, have them use the same URL to save on storage space.

  • A tiny (<10MB) Docker image (jasonwhite0/rudolfs).

The back-end storage code is very modular and composable. PRs for implementing other storage back-ends are welcome. If you begin working on this, please let us know by submitting an issue.

Non-Features

  • There is no client authentication. This is meant to be run in an internal network with clients you trust, not on the internet with malicious actors.

Running It

Generate an encryption key

All LFS objects are encrypted with the xchacha20 symmetric stream cipher. You must generate a 32-byte encryption key before starting the server.

Generating a random key is easy:

openssl rand -hex 32

Keep this secret and save it in a password manager so you don't lose it. We will pass this to the server below.

Note:

  • If the key ever changes, all existing LFS objects will become garbage. When the Git LFS client attempts to download them, the SHA256 verification step will fail.
  • LFS objects in both the cache and in permanent storage are encrypted. However, objects are decrypted before being sent to the LFS client, so take any necessary precautions to keep your intellectual property safe.

Development

For testing during development, it is easiest to run it with Cargo. Create a file called test.sh (this path is already ignored by .gitignore):

# Your AWS credentials.
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
export AWS_DEFAULT_REGION=us-west-1

# Change this to the output of `openssl rand -hex 32`.
KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

cargo run -- \
  --cache-dir cache \
  --host localhost:8080 \
  --max-cache-size 10GiB \
  --key $KEY \
  s3 \
  --bucket foobar

Note: Always use a different S3 bucket, cache directory, and encryption key than what you use in your production environment.

Production

To run in a production environment, it is easiest to use docker-compose:

  1. Create a .env file next to docker-compose.yml with the configuration variables:

    AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXQ
    AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    AWS_DEFAULT_REGION=us-west-1
    LFS_ENCRYPTION_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    LFS_S3_BUCKET=my-bucket
    LFS_MAX_CACHE_SIZE=10GB
    
  2. Use the provided docker-compose.yml file to run a production environment:

    docker-compose up -d
    
  3. [Optional]: It is best to use nginx as a reverse proxy for this server. Use it to enable TLS. How to configure this is better covered by other tutorials on the internet.

Note:

  • A bigger cache is (almost) always better. Try to use ~85% of the available disk space.
  • The cache data is stored in a Docker volume named rudolfs_data. If you want to delete it, run docker volume rm rudolfs_data.

AWS Credentials

AWS credentials must be provided to the server so that it can make requests to the S3 bucket specified on the command line (with --s3-bucket).

Your AWS credentials will be searched for in the following order:

  1. Environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  2. AWS credentials file. Usually located at ~/.aws/credentials.
  3. IAM instance profile. Will only work if running on an EC2 instance with an instance profile/role.

The AWS region is read from the AWS_DEFAULT_REGION or AWS_REGION environment variable. If it is malformed, it will fall back to us-east-1. If it is not present it will fall back on the value associated with the current profile in ~/.aws/config or the file specified by the AWS_CONFIG_FILE environment variable. If that is malformed or absent it will fall back to us-east-1.

Client Configuration

Add .lfsconfig to the root of your Git repository:

[lfs]
url = "http://gitlfs.example.com:8080/api/my-org/my-project"

Optionally, I also recommend changing these global settings to speed things up:

# Increase the number of worker threads
git config --global lfs.concurrenttransfers 64

# Use a global LFS cache to make re-cloning faster
git config --global lfs.storage ~/.cache/lfs

License

MIT License

Thanks

This was developed at Environmental Systems Research Institute (Esri) who have graciously allowed me to retain the copyright and publish it as open source software.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].