gaul / Are We Consistent Yet
Licence: apache-2.0
Compare eventual consistency of object stores
Stars: ✭ 157
Programming Languages
java
68154 projects - #9 most used programming language
Projects that are alternatives of or similar to Are We Consistent Yet
S3proxy
Access other storage backends via the S3 API
Stars: ✭ 952 (+506.37%)
Mutual labels: azure, s3, aws-s3, google-cloud-storage
Goofys
a high-performance, POSIX-ish Amazon S3 file system written in Go
Stars: ✭ 3,932 (+2404.46%)
Mutual labels: s3, aws-s3, google-cloud-storage
Drone Cache
A Drone plugin for caching current workspace files between builds to reduce your build times
Stars: ✭ 194 (+23.57%)
Mutual labels: azure, s3, google-cloud-storage
Mc
MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.
Stars: ✭ 1,962 (+1149.68%)
Mutual labels: s3, aws-s3, google-cloud-storage
Thanos
Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
Stars: ✭ 9,820 (+6154.78%)
Mutual labels: s3, google-cloud-storage
Radosgw Admin4j
A Ceph Object Storage Admin SDK / Client Library for Java ✨🍰✨
Stars: ✭ 50 (-68.15%)
Mutual labels: s3, aws-s3
0x4447 product s3 email
📫 A serverless email server on AWS using S3 and SES
Stars: ✭ 2,905 (+1750.32%)
Mutual labels: s3, aws-s3
Kafka Connect Storage Cloud
Kafka Connect suite of connectors for Cloud storage (Amazon S3)
Stars: ✭ 153 (-2.55%)
Mutual labels: s3, aws-s3
Awslib scala
An idiomatic Scala wrapper around the AWS Java SDK
Stars: ✭ 20 (-87.26%)
Mutual labels: s3, aws-s3
S3 Permission Checker
Check read, write permissions on S3 buckets in your account
Stars: ✭ 18 (-88.54%)
Mutual labels: s3, aws-s3
Nestjs Multer Extended
💪 Extended MulterModule for NestJS with flexible S3 upload and helpful features
Stars: ✭ 99 (-36.94%)
Mutual labels: s3, aws-s3
Rumble
⛈️ Rumble 1.11.0 "Banyan Tree"🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more
Stars: ✭ 58 (-63.06%)
Mutual labels: azure, s3
Simpleupload
Simple upload system in PHP, compatible with AWS S3, Dropbox, Azure and others.
Stars: ✭ 85 (-45.86%)
Mutual labels: azure, aws-s3
Foundatio
Pluggable foundation blocks for building distributed apps.
Stars: ✭ 1,365 (+769.43%)
Mutual labels: azure, s3
Rome
Carthage cache for S3, Minio, Ceph, Google Storage, Artifactory and many others
Stars: ✭ 724 (+361.15%)
Mutual labels: s3, aws-s3
Migrate
Database migrations. CLI and Golang library.
Stars: ✭ 7,712 (+4812.1%)
Mutual labels: aws-s3, google-cloud-storage
Are We Consistent Yet?
Observed and documented eventual consistency of object stores, e.g., Amazon S3, OpenStack Swift.
What is eventual consistency?
Traditional systems provide strong consistency, where clients can immediately view updates. Some distributed systems relax their consistency model to allow greater availability or better performance. Eventual consistency manifests itself to clients as stale views of data.
Observed consistency
We ran a test in which we wrote (i.e., create, update, delete) an object and then attempted to read the object. Across many trials, we count the number of times the object was not immediately available. When the object is not immediately found, it's an occurrence of observed eventual consistency.
Observed instances of eventual consistency with a number of operations during tests around 17 December 2014 with a 1-byte object size:
| Provider | read after create | read after delete | read after overwrite | list after create | list after delete | number of operations |
|---|---|---|---|---|---|---|
| Amazon S3 (us-standard) | * | * | * | 16 | * | 100,000 |
| Amazon S3 (us-standard†) | 16 | 8 | 12 | 8 | 3 | 1,000 |
| Amazon S3 (us-west) | - | 1 | 1 | * | * | 100,000 |
| Ceph (DreamObjects) | - | - | - | - | - | 1,000 |
| Google Cloud Storage | - | - | - | 2 | 2 | 1,000 |
| Microsoft Azure Storage | - | - | - | - | - | 1,000 |
| OpenStack Swift (Rackspace) | - | 17 | 3 | 30 | 20 | 1,000 |
Legend:
- - zero observed instances
- * zero observed instances but expect non-zero
- † writing to N. Virginia and reading from Pacific Northwest
Documented consistency
- Amazon S3 - buckets in the US Standard region provide eventual consistency. Buckets in all other regions provide read-after-write consistency for PUTs of new objects and eventual consistency for overwrite PUTs and DELETEs.
- Ceph - provides well-defined safety semantics and strong consistency guarantees.
- Google Cloud Storage - provides strong global consistency for all read-after-write, read-after-update, and read-after-delete operations, including both data and metadata. Bucket and object listing are also strongly consistent.
- Microsoft Azure Storage - was designed to embrace a strong consistency model which guarantees that when the Storage service commits a data insert or update operation all further accesses to that data will see the latest update.
- OpenStack Swift - For example, suppose a container server is under load and a new object is put in to the system. The object will be immediately available for reads as soon as the proxy server responds to the client with success. However, the container server did not update the object listing, and so the update would be queued for a later update. Container listings, therefore, may not immediately contain the object. Additional reference.
References
- Apache jclouds - provides object storage support
- A Middleware Guaranteeing Client-Centric Consistency on Top of Eventually Consistent Datastores
- Benchmarking Eventual Consistency: Lessons Learned from Long-Term Experimental Studies
- Eventual Consistency: How soon is eventual?
- Eventual Consistency Today: Limitations, Extensions, and Beyond
- Probabilistically Bounded Staleness - includes discussion of Riak and Cassandra
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].