All Projects â†’ Snowtrack â†’ Snowfs

Snowtrack / Snowfs

Licence: mit
SnowFS - a fast, scalable version control file storage for graphic files 🎨

Programming Languages

typescript
32286 projects

Projects that are alternatives of or similar to Snowfs

snowfs
SnowFS - a fast, scalable version control file storage for graphic files 🎨
Stars: ✭ 1,101 (+86.61%)
Mutual labels:  storage, version-control, version-manager
Nhversion
NHVersion for version your api
Stars: ✭ 13 (-97.8%)
Mutual labels:  version-manager, version-control
Nodist
Natural node.js and npm version manager for windows.
Stars: ✭ 1,276 (+116.27%)
Mutual labels:  version-manager, version-control
infinitree
Scalable and encrypted embedded database with 3-tier caching
Stars: ✭ 80 (-86.44%)
Mutual labels:  storage, version-control
Sparkleshare
Share and collaborate by syncing with any Git repository instantly. Linux, macOS, and Windows.
Stars: ✭ 4,661 (+690%)
Mutual labels:  version-control
Csi Digitalocean
A Container Storage Interface (CSI) Driver for DigitalOcean Block Storage
Stars: ✭ 452 (-23.39%)
Mutual labels:  storage
Jsstore
A complete IndexedDB wrapper with SQL like syntax.
Stars: ✭ 430 (-27.12%)
Mutual labels:  storage
Kinto
A generic JSON document store with sharing and synchronisation capabilities.
Stars: ✭ 4,150 (+603.39%)
Mutual labels:  storage
Filegator
Powerful Multi-User File Manager
Stars: ✭ 587 (-0.51%)
Mutual labels:  storage
S5cmd
Parallel S3 and local filesystem execution tool.
Stars: ✭ 565 (-4.24%)
Mutual labels:  storage
Html
HTML Standard
Stars: ✭ 5,217 (+784.24%)
Mutual labels:  storage
Sleekdb
Pure PHP NoSQL database with no dependency. Flat file, JSON based document database.
Stars: ✭ 450 (-23.73%)
Mutual labels:  storage
Rxfirebase
Rxjava 2.0 wrapper on Google's Android Firebase library.
Stars: ✭ 509 (-13.73%)
Mutual labels:  storage
Imagestore
Open source google photos alternative!
Stars: ✭ 429 (-27.29%)
Mutual labels:  storage
Temporary Directory
A simple class to work with a temporary directory
Stars: ✭ 567 (-3.9%)
Mutual labels:  storage
Cortx
CORTX Community Object Storage is 100% open source object storage uniquely optimized for mass capacity storage devices.
Stars: ✭ 426 (-27.8%)
Mutual labels:  storage
Offline Plugin
Offline plugin (ServiceWorker, AppCache) for webpack (https://webpack.js.org/)
Stars: ✭ 4,444 (+653.22%)
Mutual labels:  storage
Cpm.cmake
📦 CMake's missing package manager. A small CMake script for setup-free, cross-platform, reproducible dependency management.
Stars: ✭ 560 (-5.08%)
Mutual labels:  version-manager
Jupytext
Jupyter Notebooks as Markdown Documents, Julia, Python or R scripts
Stars: ✭ 4,969 (+742.2%)
Mutual labels:  version-control
Vue Ls
💥 Vue plugin for work with local storage, session storage and memory storage from Vue context
Stars: ✭ 468 (-20.68%)
Mutual labels:  storage

snowfs

SnowFS - a fast, scalable version control file storage for graphic files

release Coverage Status Build and Test MIT Licence

Overview

SnowFS is a lightweight command-line application and library with a focus on binary file versioning. It is made for the graphics industry and was initially developed for Snowtrack.

Disclaimer: This project is in alpha state and actively developed. Do not use this yet in a production environment or without backups.

terminal

Feature highlights

  • Supports Branches

  • Asynchronous File Hashing

  • Project open to file-content awareness (e.g: *.psd, *.blend, *.c4d, ..)

  • Super-fast-detection of modifications in large binaries

  • Support for instant snapshots**

  • Support for instant rollback**

  • Support for files bigger >4TB

  • Block-cloning and Copy-on-Write support for APFS, ReFS and Btrfs

  • Support for removing single versions and/or binaries

  • Primarily I/O bound through libuv

  • Feature XYZ made by you!

** If the underlying filesystem supports it (e.g. APFS, ReFS and Btrfs)

Why not Git/Git-LFS, libgit2, or SVN?

First and foremost, the implementations of Git - namely Git/Git-LFS and libgit2 are excellent implementations of version control systems. But due to their focus on the software development lifecycle they are not suitable to version binaries or graphic files. SnowFS addresses the technical challenges for graphic files by its core design.

Git/Git-LFS:

Advantages:

  • Support on all major platforms
  • Supported by hosting platforms like GitHub, GitLab, and BitBucket.
  • Fast diff-operation for text-files

Disadvantages:

  • If not properly tracked, binaries become accidentally part of "base" history
  • Removing older commits is cumbersome due to Gits commit hashing integrity
  • Complicated rewriting history procedure
  • Issues with binaries >4GB on Windows as reported here, here, and here
  • Slow in binary modification detection
  • Git uses a restrictive license

libgit2

Advantages:

  • Faster zipping, packing, and delta-compression than the reference implementation Git
  • Supports custom backends

Disadvantages:

  • No native support for Git-LFS without custom backends
  • Custom backends break compatibility with Git

TypeScript / C++ backport

SnowFS is currently written in TypeScript. It is a great language to write powerful and performant I/O bound prototypes. We are aware of the demand for a C++ implementation, and due to our roots as C++ developers, we are very interested in a backport as well. It would make SnowFS easier to integrate into other projects. If you have comments, ideas or recommendations, please let us know.

Running benchmarks

We have also implemented a comparison benchmark between SnowFS vs. git-lfs. After executing the build instructions for a development build, the benchmarks can be executed with the following command:

$ npm run benchmarks

Example run on a Macbook Pro (2020) with an APFS formatted SSD to check-in, delete and restore a 4GB Photoshop File.

...
git lfs track *.psd
git add texture.psd: 20164ms
snow add texture.psd: 4596ms
git rm texture.psd: 575ms
snow rm texture.psd: 111ms
git checkout HEAD~1: 9739ms
snow checkout HEAD~1: 1ms <-- Yeap!

Examples

Code

You can find the best and up-to-date code examples in the test/ directory. Given below these are simply "Hello World!" examples to get you started.

import * as fse from "fs-extra";

import { join } from "path";
import { Index } from "./src";
import { Repository } from "./src/repository";

export async function main() {
  let repo: Repository;
  let index: Index;
  const repoPath = "/path/to/a/non/existing/directory";
  Repository.initExt(repoPath)
    .then((repoResult: Repository) => {
      return fse.copyFile("/path/to/texture.psd", join(repoPath, "texture.psd"));
    })
    .then(() => {
      index.addFiles(["texture.psd"]);
      return index.writeFiles();
    })
    .then(() => {
      return repo.createCommit(index, "This is my first commit");
    });
}

main();

Command line interface

The CLI of SnowFS offers some basic functionality and is subject to enhancements. See Report #90

release

$ snow init foo
$ cp /path/to/texture.psd foo/texture.psd
$ cd foo
$ snow add .
$ snow commit -m "My first texture"
$ snow log
$ snow checkout -b MyNewBranch
$ snow log

Build Instructions

  1. To build SnowFS install node.js for your specific platform.

    1.1. On Windows, Visual Studio is required. We recommend the Visual Studio 2019 Community Edition. During the installation, please enable Desktop development with C++.

    1.2. On MacOS, XCode is required.

  2. To build a development build execute:

$ git clone https://github.com/Snowtrack/snowfs.git
$ cd snowfs.git
$ npm install
$ npm run ava
  1. To build a production build including the CLI, execute the commands above, and continue with the commands below:
$ npm run tsc
$ npm run build
$ cd dist/out-tsc

How To Debug

For the development of SnowFS we recommend VSCode. The repository contains a launch.json file with pre-defined runner configurations. For more information, please visit this pull-request.

Versioning

Starting with version 1.0.0 SnowFS follows the semantic versioning scheme. The API change and backward compatibility rules are those indicated by SemVer.

Licensing

SnowFS is licensed under the MIT license, please review the LICENSE file. Excluded from the license are images, artworks, and logos. Please file a request by mail, if you have any questions.

Community

Other resources

The tests and benchmarks also serve as API specification and usage examples.

Supported Platforms

Currently, Windows, macOS, and Linux are supported. SnowFS works on plain filesystems like FAT, NTFS, HFS+ and has extended support for APFS, ReFS and Btrfs.

How can I contribute?

See the guidelines for contributing.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].