All Projects → beyondstorage → go-storage

beyondstorage / go-storage

Licence: Apache-2.0 license
A vendor-neutral storage library for Golang: Write once, run on every storage service.

Programming Languages

go
31211 projects - #10 most used programming language
Makefile
30231 projects

Projects that are alternatives of or similar to go-storage

Go Storage
An application-oriented unified storage layer for Golang.
Stars: ✭ 87 (-77.52%)
Mutual labels:  files, dropbox, storage, fs, s3, cloud-storage
Goofys
a high-performance, POSIX-ish Amazon S3 file system written in Go
Stars: ✭ 3,932 (+916.02%)
Mutual labels:  s3, cloud-storage, gcs
storage
Go library providing common interface for working across multiple cloud storage backends
Stars: ✭ 154 (-60.21%)
Mutual labels:  storage, s3, gcs
Rclone
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Stars: ✭ 30,541 (+7791.73%)
Mutual labels:  dropbox, s3, cloud-storage
Cloudexplorer
Cloud Explorer
Stars: ✭ 170 (-56.07%)
Mutual labels:  storage, s3, cloud-storage
rclone-drive
☁️Simple web cloud storage based on rclone, transform cloud storage (s3, google drive, one drive, dropbox) into own custom web-based storage
Stars: ✭ 30 (-92.25%)
Mutual labels:  dropbox, s3, cloud-storage
Cakephp File Storage
Abstract file storage and upload plugin for CakePHP. Write to local disk, FTP, S3, Dropbox and more through a single interface. It's not just yet another uploader but a complete storage solution.
Stars: ✭ 202 (-47.8%)
Mutual labels:  dropbox, storage, s3
Space Daemon
The Space Daemon packages together IPFS, Textile Threads/Buckets, and Textile Powergate (Filecoin*) into one easy to install Daemon to make it easy to build peer to peer and privacy focused apps.
Stars: ✭ 151 (-60.98%)
Mutual labels:  files, storage
matched
Glob matching with support for multiple patterns and negation. Use `~` in cwd to find files in user home, or `@` for global npm modules.
Stars: ✭ 25 (-93.54%)
Mutual labels:  files, fs
esop
Cloud-enabled backup and restore tool for Apache Cassandra
Stars: ✭ 40 (-89.66%)
Mutual labels:  storage, s3
dropbox-fs
📦 Node FS wrapper for Dropbox
Stars: ✭ 35 (-90.96%)
Mutual labels:  dropbox, fs
Write Files Atomic
Write many files atomically
Stars: ✭ 126 (-67.44%)
Mutual labels:  files, fs
File Storage
File storage abstraction for Yii2
Stars: ✭ 116 (-70.03%)
Mutual labels:  files, storage
kafka-connect-fs
Kafka Connect FileSystem Connector
Stars: ✭ 107 (-72.35%)
Mutual labels:  files, s3
Wsend
wsend: The opposite of wget
Stars: ✭ 64 (-83.46%)
Mutual labels:  files, storage
cottoncandy
sugar for s3
Stars: ✭ 33 (-91.47%)
Mutual labels:  s3, cloud-storage
hub
Public reusable components for Polyaxon
Stars: ✭ 8 (-97.93%)
Mutual labels:  s3, gcs
backblaze
Backblaze.Agent is a high-performance .NET Core implementation of the Backblaze B2 Cloud Storage API.
Stars: ✭ 32 (-91.73%)
Mutual labels:  s3, cloud-storage
go-drive
A simple cloud drive mapping web app supports local, FTP/SFTP, S3, OneDrive, WebDAV, Google Drive.
Stars: ✭ 184 (-52.45%)
Mutual labels:  s3, cloud-storage
SaorTech-cloud-services
A range of scripts to provision and configure open source cloud services.
Stars: ✭ 23 (-94.06%)
Mutual labels:  dropbox, cloud-storage

go-storage

Website | Documentation | Community

Go dev License

Matrix Slack Telegram

Build Test Cross Build Unit Test

A vendor-neutral storage library for Golang.

Vision

Write once, run on every storage service.

Goal

  • Vendor agnostic
  • Production ready
  • High performance

Examples

package main

import (
    "log"

    "github.com/beyondstorage/go-storage/v5/services"
    "github.com/beyondstorage/go-storage/v5/types"

    // Add fs support
    _ "github.com/beyondstorage/go-storage/services/fs/v4"
    // Add s3 support
    _ "github.com/beyondstorage/go-storage/services/s3/v3"
    // Add gcs support
    _ "github.com/beyondstorage/go-storage/services/gcs/v3"
    // Add azblob support
    _ "github.com/beyondstorage/go-storage/services/azblob/v3"
    // More support could be found under BeyondStorage.
    _ "github.com/beyondstorage/go-storage/services/xxx" 
)

func main() {
    // Init a Storager from connection string. 
    store, err := services.NewStoragerFromString("s3://bucket_name/path/to/workdir")
    if err != nil {
        log.Fatalf("service init failed: %v", err)
    }

    // Write data from io.Reader into hello.txt
    n, err := store.Write("hello.txt", r, length)

    // Read data from hello.txt to io.Writer
    n, err := store.Read("hello.txt", w)

    // Stat hello.txt to check existence or get its metadata
    o, err := store.Stat("hello.txt")

    // Use object's functions to get metadata
    length, ok := o.GetContentLength()
    
    // List will create an iterator of object under path.
    it, err := store.List("path")
    
    for {
    	// Use iterator.Next to retrieve next object until we meet IterateDone.
    	o, err := it.Next()
    	if errors.Is(err, types.IterateDone) {
    		break
        }
    }

    // Delete hello.txt
    err = store.Delete("hello.txt")
}

More examples could be found at go-storage-example.

Features

Widely native services support

16 stable services that have passed all integration tests.

3 beta services that implemented required functions, but not passed integration tests.

4 alpha services that still under development.

More service ideas could be found at Service Integration Tracking.

Complete and easily extensible interface

Basic operations

  • Metadata: get Storager metadata
meta := store.Metadata()
_ := meta.GetWorkDir() // Get object WorkDir
_, ok := meta.GetWriteSizeMaximum() // Get the maximum size for write operation
  • Read: read Object content
// Read 2048 byte at the offset 1024 into the io.Writer.
n, err := store.Read("path", w, pairs.WithOffset(1024), pairs.WithSize(2048))
  • Write: write content into Object
// Write 2048 byte from io.Reader
n, err := store.Write("path", r, 2048)
  • Stat: get Object metadata or check existences
o, err := store.Stat("path")
if errors.Is(err, services.ErrObjectNotExist) {
	// object is not exist
}
length, ok := o.GetContentLength() // get the object content length.
  • Delete: delete an Object
err := store.Delete("path") // Delete the object "path"
  • List: list Object in given prefix or dir
it, err := store.List("path")
for {
	o, err := it.Next()
	if err != nil && errors.Is(err, types.IterateDone) {
        // the list is over 
    }
    length, ok := o.GetContentLength() // get the object content length.
}

Extended operations

  • Copy: copy a Object inside storager
err := store.(Copier).Copy(src, dst) // Copy an object from src to dst.
  • Move: move a Object inside storager
err := store.(Mover).Move(src, dst) // Move an object from src to dst.
  • Reach: generate a public accessible url to an Object
url, err := store.(Reacher).Reach("path") // Generate an url to the object.
  • Dir: Dir Object support
o, err := store.(Direr).CreateDir("path") // Create a dir object.

Large file manipulation

  • Multipart: allow doing multipart uploads
ms := store.(Multiparter)

// Create a multipart object.
o, err := ms.CreateMultipart("path")
// Write 1024 bytes from io.Reader into a multipart at index 1
n, part, err := ms.WriteMultipart(o, r, 1024, 1)
// Complete a multipart object.
err := ms.CompleteMultipart(o, []*Part{part})
  • Append: allow appending to an object
as := store.(Appender)

// Create an appendable object.
o, err := as.CreateAppend("path")
// Write 1024 bytes from io.Reader.
n, err := as.WriteAppend(o, r, 1024)
// Commit an append object.
err = as.CommitAppend(o)
  • Block: allow combining an object with block ids
bs := store.(Blocker)

// Create a block object.
o, err := bs.CreateBlock("path")
// Write 1024 bytes from io.Reader with block id "id-abc"
n, err := bs.WriteBlock(o, r, 1024, "id-abc")
// Combine block via block ids.
err := bs.CombineBlock(o, []string{"id-abc"})
  • Page: allow doing random writes
ps := store.(Pager)

// Create a page object.
o, err := ps.CreatePage("path")
// Write 1024 bytes from io.Reader at offset 2048
n, err := ps.WritePage(o, r, 1024, 2048)

Comprehensive metadata

Global object metadata

  • id: unique key in service
  • name: relative path towards service's work dir
  • mode: object mode can be a combination of read, dir, part and more
  • etag: entity tag as defined in rfc2616
  • content-length: object's content size.
  • content-md5: md5 digest as defined in rfc2616
  • content-type: media type as defined in rfc2616
  • last-modified: object's last updated time.

System object metadata

Service system object metadata like storage-class and so on.

o, err := store.Stat("path")

// Get service system metadata via API provides by go-service-s3.
om := s3.GetObjectSystemMetadata(o)
_ = om.StorageClass // this object's storage class
_ = om.ServerSideEncryptionCustomerAlgorithm // this object's sse algorithm

Strong Typing Everywhere

Self maintained codegen definitions helps to generate all our APIs, pairs and metadata.

Generated pairs which can be used as API optional arguments.

func WithContentMd5(v string) Pair {
    return Pair{
        Key:   "content_md5",
        Value: v,
    }
}

Generated object metadata which can be used to get content md5 from object.

func (o *Object) GetContentMd5() (string, bool) {
    o.stat()
    
    if o.bit&objectIndexContentMd5 != 0 {
        return o.contentMd5, true
    }
    
    return "", false
}

Server-Side Encrypt

Server-Side Encrypt supports via system pair and system metadata, and we can use Default Pairs to simplify the job.

func NewS3SseC(key []byte) (types.Storager, error) {
    defaultPairs := s3.DefaultStoragePairs{
        Write: []types.Pair{
            // Required, must be AES256
            s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
            // Required, your AES-256 key, a 32-byte binary value
            s3.WithServerSideEncryptionCustomerKey(key),
        },
        // Now you have to provide customer key to read encrypted data
        Read: []types.Pair{
            // Required, must be AES256
            s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),
            // Required, your AES-256 key, a 32-byte binary value
            s3.WithServerSideEncryptionCustomerKey(key),
        }}
    
    return s3.NewStorager(..., s3.WithDefaultStoragePairs(defaultPairs))
}

Sponsor

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].