username=jmxusername
password=jmxpassword
keystorePassword=keystorepassword
truststorePassword=truststorepassword

The resolution of credentials for S3 uses the same resolution mechanism as the official AWS S3 client uses. The most notable fact is that if no credentials are set explicitly, it will try to resolve them from environment properties of the node it runs on. If that node runs in AWS EC2, it will resolve them by help of that particular instance.

S3 connectors will expect to find environment properties AWS_ACCESS_KEY_ID and AWS_SECRET_KEY. They will also accept AWS_REGION and AWS_ENDPOINT environment properties—however they are not required. If AWS_ENDPOINT is set, AWS_REGION has to be set too.

S3 currently supports two different addressing models: path-style and virtual-hosted style (https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAPI.html).

Esop supports different S3 providers and applies the following default addressing models.

Providing the AWS_ENABLE_PATH_STYLE_ACCESS environment variable with true or false overrides this default setting. Note that this applies to each provider, except when running in Kubernetes.

|provider|model| |Minio|path|

The communication with S3 might be insecure, this is controlled by --insecure-http flag on the command line. By default, it uses HTTPS.

It is possible to connect to S3 via proxy; please consult "--use-proxy" flag and "--proxy-*" family of settings on command line.

Azure module expects AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY environment variables to be set.

GCP module expects GOOGLE_APPLICATION_CREDENTIALS environment property or google.application.credentials to be set with the path to service account credentials.

Oracle module behaves same way as S3 when it comes to credentials.

CEPH module uses Amazon S3 driver for Ceph Object Gateway. Credentials-wise, it behaves same way as "normal" S3. You are required to set endpoint to AmazonS3 client. In that case, be sure AWS_ENDPOINT environment property is set or awsendpoint property in Kubernetes secret is specified. You need to provide typical access key and secret key too. Please consult the following section to know more about Kubernetes-related authentication properties resolution. Setting protocol to HTTP might be achieved similarly as for normal S3 module, by specifying --insecure-http flag.

minio is alias of oracle. Oracle nor Minio uses path-style requests which S3 module does not.

If this tooling is run in the context of Kubernetes, we need to inject these credentials dynamically upon every request. If these credentials are not set statically, e.g. as environment or system properties, we may have an application like Cassandra Sidecar which resolves these credentials on every backup or restore request so they may change over time by Kubernetes operators (as person). By dynamic injecting, we are separating the lifecycle of a credential from the lifecycle of a backup/restore/Sidecar application.

Credentials are stored as a secret. Namespace to read that secret from is specified by flag --k8s-namespace and the secret to read credentials from is specified by flag --k8s-secret-name. If namespace flag is not used, it defaults to default. If the secret name is not used, it is resolved as cassandra-backup-restore-secret-cluster-{cluterId} where clusterId is taken from cluster name in --storage-location.

The secret has to contain these fields:

Of course, if we do not plan to use other storage providers, feel free to omit the properties for them.

For S3, only the secret key and access key are required.

The fact that the code is running in the context of Kubernetes is derived from two facts:

• there are environment properties KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT in a respective container this tool is invoked in

• This tool runs outside of Kubernetes but as a client meaning it will resolve credentials from there but it does not run in any container. This is helpful for example during tests where we do not run it inside Kubernetes but we want to be sure that the logic dealing with the credentials resolution works properly. This is controlled by system property kubernetes.client which is by default false.

There might be the third (rather special) case—we want to run this tool in Kubernetes (so env properties would be there) but we want to run it as a client. Normally, the first condition would be fulfilled. There is a property called pretend.not.running.in.kubernetes, defaults to false. If set to true, even we run our tool in Kubernetes, it will act as a client, so it will not retrieve credentials from Kubernetes secret but from system and environment variables.

A manifest file is uploaded with all data. It contains all information necessary to restore that snapshot.

Manifest name has this format: snapshot3-f1159959-593d-33d1-9ade-712ea55b31ef-1600645759830.json

• snapshot3—name of snapshot used during a backup

• f1159959-593d-33d1-9ade-712ea55b31ef schema version of Cassandra

• 1600645759830 timestamp when that snapshot/backup was taken

The content of a manifest file looks like this:

A manifest maps all resources related to a snapshot, their size as well as type (FILE or CQL_SCHEMA). It holds all schema content in a respective file too, so we do not need to read/parse the schema file as it is already a part of the manifest.

Upon restore, this file is read into its Java model and enriched by setting a path where each manifest entry should be physically located on disk as we need to remove part of the file where a hash is specified. It is also possible to filter this manifest in such a way that we might backup 5 tables, but we want to restore only 2 of them so the other three tables would not be downloaded at all.

Topology file is uploaded during a backup as well. It is uploaded into a bucket’s topology directory in root. A topology file is provided not only as a reference to see what the topology was upon backup, but it also helps Instaclustr Cassandra operator to resolve which node it should download data for.

If we are restoring a cluster from scratch and all we have is its former hostname, we need to know what was the node’s id (nodeId below) because that id signifies which directory its data is stored in. When Instaclustr Cassandra operator restores a cluster from scratch, it knows a name of a pod (its hostname) but it does not know the id to load data from. The storage location upon a restore looks like s3://bucket/test-cluster/dc1/cassandra-test-cluster-dc1-west1-b-0. Internally, based on a snapshot and schema, we resolve the correct topology file and we filter its content to see which node starts on that hostname so we use, in this case, nodeId 8619f3e2-756b-4cb1-9b5a-4f1c1aa49af6 upon restoration. Storage location flag is then updated to use this node, so it will look like s3://bucket/test-cluster/dc1/8619f3e2-756b-4cb1-9b5a-4f1c1aa49af6.

A name of a topology file has this format clusterName-snapshotName-schemaVersion-timestamp. This uniquely identifies a topology in time.

Lets say we have done a backup against a node, multiple times, where some snapshot names were the same and schema version was the same too, for some cases we will have these manifests in a bucket:

Which manifest will be resolved when we use snapshot1 as --snapshot-tag?

If there are multiple manifests starting with same snapshot tag and having same schema version, in this particular case, it will pick the one with timestamp 1600645217800 as the latest manifest wins.

You may specify --snapshot-tag as snapshot1-f1159959-593d-33d1-9ade-712ea55b31ef or even full version with timestamp. The longest prefix wins and when there are multiple manifests resolved, the latest wins.

In case we have the same snapshot but different schema, only the snapshot name and schema version will be enough, not the snapshot name alone.

By this logic, we are preventing the situation where two operators (as a person) will do two backups with the same snapshots against a node on the same schema version—the only information which makes these two requests unique is the timestamp. However, we may use just the same snapshot name (for practical reasons not recommended) and all would work just fine.

The same resolution logic holds for topology file resolution—the longest prefix wins and it has to be uniquely filtered.

Upon backup, the schema version is determined by calling respective JMX method. The user does not have to provide it on his own. On the other hand, the second way how to resolve the problems above during restoration is to specify --exactSchemaVersion flag. When set, it will try to filter only manifests which were done on the same schema version as a current node runs on. The last option is to use --schema-version option (in connection with --exact-schema-version) with the schema version manually.

It is possible to work with a Cassandra node which has data in multiple locations, not only in one, as data_files_directories in cassandra.yaml is an array.

In order to point backup or restore procedures to multiple data directories, there is a flag called --data-dir. This flag can be set multiple times - each one pointing to different data directory, as it is set in cassandra.yaml.

Upon backup, files of all SSTables across all directories are uploaded to a remote location. However, upon restore, they are not necessarily put into the same directories.

For in-place restoration strategy, SSTables are dispersed among all data directories in a round-robin fashion.

For hard-linking strategy, it is logically same as for in-place, SSTables are again dispersed among all data directories with any signifant order.

For importing strategy, Esop does not control where SSTables will be put at all as this is delegated to imporing mechanism of Cassandra itself so the support of multiple data directories is there out of the box.

The anatomy of a backup is quite simple. The successful invocation of backup sub-command will do the following:

1. Checks if a remote bucket for whatever storage provider exists, and will optionally create it if it doesn’t (consult command line for help on how to achieve that). If a bucket does not exist and we are not allowed to create it automatically—the backup will fail.

2. Takes tokens of a respective node via JMX. Tokens are necessary for cases when we want to restore into a completely empty node. If we downloaded all data but tokens would be autogenerated, the data that node is supposed to serve would not match tokens that node is using.

3. Takes a snapshot of respective entities—either keyspaces or tables. It is not possible to mix keyspaces and some tables, it is either keyspace(s) or tables. This is inherited from the fact that Cassandra JMX API is designed that way. nodetool snapshot also permits us to specify entities to backup either as ks1,ks2,ks3 or ks1.t1,ks1.t2,ks2.t3 and we copy this behaviour here. The name of snapshot is auto generated when not specified via command line.

4. Creates internal mapping of snapshot to files it should upload.

5. Uploads SSTables and helper files to remote storage—only files which are not uploaded. By doing this, we will not "over-upload" as an SSTable is an immutable construct, so there is no need to upload what is already there. The backup procedure will check if a remote file is not there and uploads only in case it is not. Backup is doing a "hash" of an SSTable and it is uploaded under such key so it is not possible that two SSTables would be overwritten even if they are named the same as their hashes do not necessarily match.

6. The actual downloading/uploading is done in parallel—the number of simultaneous uploadings/downloadings is controlled by concurrent-connections setting which defaults to 10. It is possible to throttle the bandwidth so we do not use all available bandwidth for backups/restores so the node which might still be in operation would suffer performance-wise.

7. Writes meta-files to a remote storage—manifest and topology file (when Sidecar is used).

8. Clears taken snapshot.

As of now, a node to be backed-up has to be online because we need tokens, we need to take a snapshot, etc. and this is done via JMX. In theory we do not need a node to be online if we take a snapshot beforehand and tokens are somehow provided externally, however the current version of the tool does require it.

This tool is seamlessly integrated into Icarus which is able to do backup and restore in a distributed manner—cluster wide. Please refer to documentation of Icarus to understand what restoration phases are and what restoration strategies one might use. The very same restoration flow might be executed from CLI, Icarus just accepts a JSON payload which is a different representation of the very same data structure as the one used from command like but the functionality is completely the same.

CLI tool is not responsive to globalRequest flag in restoration/backup requests—only Sidecar can coordinate cluster-wide restoration and backup.

A restoration is a relatively more complex procedure than a backup. We have provided three strategies. You may control which strategy is used via command line.

In general, the restoration is about:

1. Downloading data from remote location

2. Making Cassandra use these files

While the first step is quite straightforward, the second depends on various factors we guide a reader through.

Restoration strategy is determined by flag --restoration-strategy-type which might be IN_PLACE, IMPORT, or HARDLINKS, case-insensitive.

In-place strategy must be used only in case a Cassandra node is down— Cassandra process does not run. This strategy will download only SSTables (and related files) which are not present locally, and it will directly download them to their respective data directories of a node. Then it will remove SSTables (and related files) which should not be there. As a backup is done against a snapshot; restore is also done from a snapshot.

Use this strategy if you want to:

• restore from an older snapshot and your node does not run

• restore from a snapshot and your node is completely empty—it was never run/its data dir is empty

• restore a cluster/node by Cassandra Operator. This feature is already fully embedded into our operator offering so one can restore whole clusters very conveniently.

In more detail, in-place strategy does the following:

1. Checks that a remote bucket to download data from exists and errors out if it does not

2. In case --resolve-host-id-from-topology flag is used, it will resolve a host to restore from topology file. This is handy for cases we want to restore e.g. in the context of Kubernetes by our operator.

3. Downloads a manifest—manifest contains the list of files which are logically related to a snapshot.

4. Filters out the files which need to be downloaded, as some files which are present locally might be also a part of a taken snapshot so we would download them unnecessarily.

5. Downloads files directly into Cassandra data dir.

6. Deletes files from data dir which should not be there.

7. Cleans data in other directories—hints, saved caches, commit logs.

8. Updates cassandra.yaml if present with auto_bootstrap: false and initial_token with tokens from manifest.

It is possible to restore not only user keyspaces and tables but system keyspaces too. This is necessary for the successful restoration of a cluster/node exactly as it was before as all system tables would be same. Normally, system keyspaces are not restored and one has to set this explicitly by --restore-system-keyspace flag.

In-place strategy uses also --restore-into-new-cluster flag. If specified, it will restore only system keyspaces needed for successful restoring (system_schema) but it will not attempt to restore anything else. In an environment like Kubernetes, we do not want to restore everything because system keyspaces contain details like tokens, peers with ips, etc. and this information is very specific to each one so we do not restore them. However, if we did not restore system_schema, the newly started node would not see the restored data as there would not be any schema. By restoring system_schema, Cassandra will detect these keyspaces and tables on the very first start.

In-place restoration might update cassandra.yaml file if found. This is done automatically upon restoration in Cassandra operator but it might be required to be done manually for other cases. By default, cassandra.yaml is not updated. The updating is enabled by setting --update-cassandra-yaml flag upon restore. It is expected that cassandra.yaml is located in a directory {cassandraConfigDirectory}/ (by default /etc/cassandra). The Cassandra configuration directory with cassandra.yaml might be changed via --config-directory flag. There are two options which are automatically changed when cassanra.yaml if found, in connection with this strategy:

• auto_bootstrap - if not found, it will be appended and set to false. If found and set to true, it will be replaced by false. If auto_bootstrap: false is already present, nothing happens.

• initial_token—set only in case it is not present cassandra.yaml. Tokens are set in order to have the node we are restoring to on the same tokens as the node we took a snapshot from.

This strategy is supposed to be executed against a running node. Hard-linking strategy downloads data from a bucket to a node’s local directory and it will make hardlinks from these files to Cassandra data dir for that keyspace/table. After hardlinks are done, it will refresh a respective table / keyspace via JMX so Cassandra will start to read from them. Afterwards, the original files are deleted.

This strategy works for Cassandra version 3 as well as for Cassandra 4.

This strategy is similar to hardlinking strategy — the node upon restoration can still run and serve other requests so a restoration process is not disruptive. Importing means that it will import downloaded SSTables via JMX directly so no hardlinks and refresh are necessary. Importing of SSTables by calling respecting JMX method was introduced in Cassandra 4 only, so this does not work against a node of version 3 or below. Keep in mind that imported SSTables are physically deleted from download directory and moved to live Cassandra data directory.

Hardlinking and importing strategy consists of phases. Each phase is done per node.

1. Cluster health check—this phase ensures that we are restoring into a healthy cluster, if any of this check is violated the restore will not proceed. We check that:

   1. A node under the restoration is in NORMAL state

   2. Each node in a cluster is `UP—the failure detector (as seen from that node) does not detect any node as failed

   3. All nodes are not in joining, leaving, moving state and all are reachable

   4. All nodes are on same schema version

2. Downloading phase—this phase will download all data necessary for the restore to happen.

3. Truncate phase—this phase will truncate all respective tables we want to restore.

4. Importing phase—for hardlinking strategy. It will do hardlinks from download directory to live Cassandra data dir; for importing strategy, it will call JMX method to import them.

5. Cleaning phase—this phase will cleanup a directory where Cassandra put truncated data; it will also delete the directory where downloaded SSTables are.

In a situation where we are restoring into a cluster of multiple nodes, the truncate operation should be executed only once against a particular node, as Cassandra will internally distribute the truncating operation to all nodes in a cluster. In other words, it is enough to truncate at one node only as data from all other nodes will be truncated too.

Downloading phase is proceeding all other phases because we want to be sure that we are truncating the data if and only if we have all data to restore from. If we truncated all data and download fails, we can not restore and the node does not contain any data to serve, rendering it useless (for that table) with some complicated procedure to recover the truncated data.

If any phases fail, all other phases fail too. Hence if we fail to download data, from an operational point of view nothing happens, as nothing was truncated and data on a running cluster were not touched. If we fail to truncate, we are still good. Once we truncate and we have all data, it is straightforward to import/hard-link data. This is the least invasive operation with a high probability of success.

It can be decided if we want to delete downloaded as well as truncated data after a restore is finished. If we plan to restore multiple times with the same data—for whatever reason— and to return back to the same snapshot, it is not desired to download all data all over again. We might just reuse them. This is controlled by flags --restoration-no-download-data and --restoration-no-delete-downloads respectively.

When a cluster we made a backup for is on the same schema at the time we want to do a restore, all is fine. However, a database schema evolves over time, columns are added or removed and we still want to be able to restore. Let’s look at this scenario:

1. create keyspace ks1 with table table1

2. insert data

3. make backup

4. alter table, add a column

5. insert data

6. restore into snapshot made in the 3rd step

Clearly, the schema we are on differs from the schema back then—there is a new column which is not present in uploaded SSTables. However, this will work, resulting in a column which is new to have all values for that column as null. This tool does not try to modify a schema itself. An operator would have to take care of this manually and such column would have to be dropped.

The opposite situation works as well:

1. create keyspace ks1 with table table1

2. insert data

3. make backup

4. alter table, drop a column

5. insert data

6. restore into snapshot made in the 3rd step

If we want to restore, we have one column less from snapshot, data will be imported but that column will just not be there.

As of now, the restore is only "forward-compatible" on a table level. If we dropped whole table and we want to restore it, this is not possible—the table has to be there already. You may recreate them by applying respective CQL create statements from the manifest before proceeding. The tool might try to create these tables beforehand as we have that CQL schema at hand, but currently it is not implemented.

$ java -jar esop.jar commitlog-backup \
  --storage-location=s3://myBucket/mycluster/dc1/node1 \
  --commit-log-dir /var/lib/cassandra/data/commitlog

archive_command=/bin/ln %path /backup/%name

$ java -jar esop.jar commitlog-backup \
  --storage-location=s3://myBucket/mycluster/dc1/node1 \
  --cl-archive=/backup

archive_command=/bin/bash /path/to/my/backup-script.sh %path %name

$!/bin/bash

java -jar esop.jar commitlog-backup \
    --storage-location=s3://myBucket/mycluster/dc1/node1 \
    --commit-log=$1

$!/bin/bash

# specifying --online will update s3://myBucket/mycluster/dc1/node1 to
# s3://myBucket/real-dc/real-dc-name/68fcbda0-442f-4ca4-86ec-ec46f2a00a71 where uuid is host id.

java -jar esop.jar commitlog-backup \
    --storage-location=s3://myBucket/mycluster/dc1/node1 \
    --commit-log=$1 \
    --online

backup \
--jmx-service 127.0.0.1:7199 \
--storage-location=s3://myBucket/mycluster/dc1/node1 \
--data-dir /my/installation/of/cassandra/data/data \
--entities=ks1,ks2 \
--snapshot-tag=mysnapshot

$ restore --data-dir /my/installation/of/cassandra/data/data \ \
          --config-directory=/my/installation/of/restored-cassandra/conf \
          --snapshot-tag=stefansnapshot" \
          --storage-location=s3://bucket-name/cluster-name/dc-name/node-id \
          --restore-system-keyspace \
          --update-cassandra-yaml=true"

backup \
--jmx-service 127.0.0.1:7199 \
--storage-location=s3://myBucket/mycluster/dc1/node1 \
--data-dir /my/installation/of/cassandra/data/data \
--entities=ks1,ks2 \
--snapshot-tag=mysnapshot

restore \
--data-dir /my/installation/of/cassandra/data/data \
--snapshot-tag=my-snapshot \
--storage-location=s3://myBucket/mycluster/dc1/node1 \
--entities=ks1,ks2 \
--restoration-strategy-type=hardlinks \
--restoration-phase-type=download, /// IMPORTANT
--import-source-dir=/where/to/put/downloaded/sstables

restore,
--data-dir /my/installation/of/cassandra/data/data \
--snapshot-tag=my-snapshot \
--storage-location=s3://myBucket/mycluster/dc1/node1 \
--entities=ks1,ks2 \
--restoration-strategy-type=hardlinks \
--restoration-phase-type=truncate \ /// IMPORTANT
--import-source-dir=/where/to/put/downloaded/sstables

restore,
--data-dir /my/installation/of/cassandra/data/data \
--snapshot-tag=my-snapshot \
--storage-location=s3://myBucket/mycluster/dc1/node1 \
--entities=ks1,ks2 \
--restoration-strategy-type=hardlinks \
--restoration-phase-type=import \ /// IMPORTANT
--import-source-dir=/where/to/put/downloaded/sstables

restore,
--data-dir /my/installation/of/cassandra/data/data \
--snapshot-tag=my-snapshot \
--storage-location=s3://myBucket/mycluster/dc1/node1 \
--entities=ks1,ks2 \
--restoration-strategy-type=hardlinks \
--restoration-phase-type=cleanup \ /// IMPORTANT
--import-source-dir=/where/to/put/downloaded/sstables

--entities="" --rename=whatever non empty  -> invalid
--entities=ks1 --rename=whatever non empty -> invalid, you can not use only a keyspace in --entities
--entities=ks1.tb1 --rename=ks1.tb2=ks1.tb2 -> invalid as "from" is not in entities
--entities=ks1.tb1 --rename=ks1.tb2=ks1.tb1 -> invalid as "to" is in entities (and from is not in entities)
--entities=ks1.tb1 --rename=ks1.tb1=ks1.tb2 -> truncate ks1.tb2 and process just ks1.tb2, k1.tb1 is not touched

--entities=ks1.tb1 --rename=ks1.tb1=ks1.tb2
--entities=ks1.tb1 --rename=ks1.tb1=ks2.tb1
--entities=ks1.tb1,ks2.tb2,ks3.tb4 --rename=ks1.tb1=ks1.tb2,ks2.tb2=ks3.tb3

{
    "rename": {
        "ks1.tb1": "ks1.tb2",
        "ks2.tb3": "ks2.tb4",
        "ks3.tb5": "ks3.tb6"
    }
}

$ commitlog-restore --commit-log-dir=/my/installation/of/restored-cassandra/data/commitlog \
                    --config-directory=/my/installation/of/restored-cassandra/conf \
                    --storage-location=s3://bucket-name/cluster-name/dc-name/node-id \
                    --commitlog-download-dir=/dir/where/commitlogs/are/downloaded \
                    --timestamp-end=unix_timestamp_of_last_transaction_to_replay

restore_directories=/home/smiklosovic/dev/instaclustr-esop/target/commitlog_download_dir
restore_point_in_time=2020\:01\:13 11\:32\:51
restore_command=cp -f %from %to