chartmuseum/storage
Go library providing a common interface for working across multiple storage backends.
Supported storage backends:
- Alibaba Cloud OSS Storage (alibaba.go)
- Amazon S3 (amazon.go)
- Baidu Cloud BOS Storage (baidu.go)
- DigitalOcean Spaces (amazon.go, using custom endpoint and us-east-1)
- etcd (etcd.go)
- Google Cloud Storage (google.go)
- Local filesystem (local.go)
- Microsoft Azure Blob Storage (microsoft.go)
- Minio (amazon.go, using custom endpoint and us-east-1)
- Netease Cloud NOS Storage (netease.go)
- Openstack Object Storage (openstack.go)
- Oracle Cloud Infrastructure Object Storage (oracle.go)
- Tencent Cloud Object Storage (tencent.go)
This code was originally part of the Helm project: ChartMuseum, but has since been released as a standalone package for others to use in their own projects.
Primary Components
Backend (interface)
Backend is a common interface that is implemented by all the supported storage backends and their associated types:
type Backend interface {
ListObjects(prefix string) ([]Object, error)
GetObject(path string) (Object, error)
PutObject(path string, content []byte) error
DeleteObject(path string) error
}Object (struct)
Object is a struct that represents a single storage object:
type Object struct {
Path string
Content []byte
LastModified time.Time
}ObjectSliceDiff (struct)
ObjectSliceDiff is a struct that represents overall changes between two Object slices:
type ObjectSliceDiff struct {
Change bool
Removed []Object
Added []Object
Updated []Object
}GetObjectSliceDiff (function)
GetObjectSliceDiff is a function that takes two Object slices, compares them, and returns an ObjectSliceDiff:
func GetObjectSliceDiff(prev []Object, curr []Object, timestampTolerance time.Duration) ObjectSliceDiffUsage
Simple example
The following is a simple program that will upload a file either to an Azure Blob Storage bucket (container) or a Google Cloud Storage bucket based on the command line options provided:
// Usage: go run example.go <cloud> <bucket> <file>
package main
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
"github.com/chartmuseum/storage"
)
type (
Uploader struct {
Backend storage.Backend
}
)
func NewUploader(cloud string, bucket string) *Uploader {
var backend storage.Backend
switch cloud {
case "azure":
backend = storage.NewMicrosoftBlobBackend(bucket, "")
case "google":
backend = storage.NewGoogleCSBackend(bucket, "")
default:
panic("cloud provider " + cloud + " not supported")
}
uploader := Uploader{Backend: backend}
fmt.Printf("uploader created (cloud: %s, bucket: %s)\n", cloud, bucket)
return &uploader
}
func (uploader *Uploader) Upload(filename string) {
basename := filepath.Base(filename)
content, err := ioutil.ReadFile(filename)
if err != nil {
panic(err)
}
err = uploader.Backend.PutObject(basename, content)
if err != nil {
panic(err)
}
fmt.Printf("%s successfully uploaded\n", basename)
}
func main() {
args := os.Args[1:]
uploader := NewUploader(args[0], args[1])
uploader.Upload(args[2])
}Example of using to upload the file index.html to an Azure bucket:
go run example.go azure mycontainer index.htmlExample of using to upload the file index.html to a Google Cloud bucket:
go run example.go google mybucket index.htmlPer backend
Each supported storage backend has its own type that implements the Backend interface.
All available types are described in detail on GoDoc.
In addition, authentication methods are based on the runtime environment and vary from cloud to cloud.