Cassandra Exporter
Cassandra exporter is a data export / import tool for cassandra that is simple to use and works for unicode and complex data types. It is developed in Javascript and the exported data is stored in JSON formatted files.
Why another tool?
Cassandra has some great tools for exporting and importing data:
- snapshots
- sstable2json
- CQL's COPY FROM/TO
But the problem is snapshots and sstable2json are not that straight forward to use. They are intended for moving large data sets and to me unnecessarily complicated to use for day to day development.
The COPY command was intended for development or moving small datasets, but is not reliable. Because it uses csv exports which breaks for complex data types and non ascii encodings if you try to import that data. So for development purposes and for moving small datasets (< few million rows per table) I needed something that works robustly and is simple to use.
Download
You can either download the compiled binary for your operating system from the releases section or if you have nodejs installed, you can use the source code directly to execute the export / import scripts.
Usage (Compiled Binary)
To export all table data from a keyspace
HOST=127.0.0.1 KEYSPACE=from_keyspace_name ./export
It will create exported json files in the data directory for each table in the keyspace.
To import all table data into a keyspace
HOST=127.0.0.1 KEYSPACE=to_keyspace_name ./import
It will process all json files in the data directory and import them to corresponding tables in the keyspace.
To export/import a single table in a keyspace
HOST=127.0.0.1 KEYSPACE=from_keyspace_name TABLE=my_table_name ./export
HOST=127.0.0.1 KEYSPACE=to_keyspace_name TABLE=my_table_name ./import
To export/import using authentication
KEYSPACE=from_keyspace_name USER=user1 PASSWORD=pa$$word ./export
KEYSPACE=to_keyspace_name USER=user1 PASSWORD=pa$$word ./import
Please note that the user requires access to the system tables in order to work properly.
To export/import using SSL/TLS
KEYSPACE=from_keyspace_name USER=user1 PASSWORD=pa$$word USE_SSL=true ./export
KEYSPACE=from_keyspace_name USER=user1 PASSWORD=pa$$word USE_SSL=true ./import
Usage (from NodeJS)
If you already have nodejs installed in your system, then you can execute using the source directly like this:
HOST=127.0.0.1 KEYSPACE=from_keyspace_name TABLE=my_table_name node export.js
HOST=127.0.0.1 KEYSPACE=from_keyspace_name TABLE=my_table_name node import.js
Usage (Docker)
The Dockerfiles provide a volume mounted at /data and expect the environment variables HOST and KEYSPACE. Dockerfile.import provides import.js functionality. Dockerfile.export provides export.js functionality. By using the -v option of docker run this provides the facility to store the output/input directory in an arbitrary location. It also allows running cassandra-export from any location. This requires Docker to be installed.
Running tests
To run a test in the tests folder, for example numbers.js, run the command node tests/numbers.js at the root of the repo. A localhost cassandra must be running.
Tests use recent node.js features and requires Node.js 8.
Note
Cassandra exporter only export / import data. It expects the tables to be present beforehand. If you need to also export schema and the indexes, then you could easily use cqlsh and the source command to export / import the schema before moving the data.
// To export keyspace schema, use cqlsh like this
cqlsh -e "DESC KEYSPACE mykeyspace" > my_keyspace_schema.cql
// To import keyspace schema open the cqlsh shell
// in the same directory of `my_keyspace_schema.cql`, then
source 'my_keyspace_schema.cql'