Implementation of the Zarr core protocol (version 2 and 3) based on xtensor
Introduction
xtensor-zarr offers an API to create and access a Zarr (v2 or v3) hierarchy in a store (locally or in the cloud), read and write arrays (in various formats) and groups in the hierarchy, and explore the hierarchy.
Installation
xtensor-zarr comes with a libxtensor-zarr-gdal shared library for accessing stores through GDAL's Virtual File System or the local file system. For all other stores, xtensor-zarr is a header-only library.
We provide a package for the mamba (or conda) package manager:
mamba install xtensor-zarr -c conda-forge-
xtensor-zarrdepends onxtensor^0.23.8,xtensor-io^0.12.7,zarray^0.0.7,nlohmann_json^3.9.1,Blosc^1.21.0,zlib^1.2.11,gdal^3.0.0andcpp-filesystem^1.3.0. -
google-cloud-cppandaws-sdk-cppare optional dependencies toxtensor-zarr.google-cloud-cppis required to access a store in Google Cloud Storage.aws-sdk-cppis required to access a store in AWS S3.
You can also install xtensor-zarr from source:
mkdir build
cd build
cmake ..
make
sudo make install
Trying it online
To try out xtensor-zarr interactively in your web browser, just click on the binder link:
Documentation
To get started with using xtensor-zarr, check out the full documentation:
http://xtensor-zarr.readthedocs.io/
Usage
// create a hierarchy on the local file system
xt::xzarr_file_system_store store("test.zr3");
auto h = xt::create_zarr_hierarchy(store);
// create an array in the hierarchy
nlohmann::json attrs = {{"question", "life"}, {"answer", 42}};
using S = std::vector<std::size_t>;
S shape = {4, 4};
S chunk_shape = {2, 2};
xt::zarray a = h.create_array(
"/arthur/dent", // path to the array in the store
shape, // array shape
chunk_shape, // chunk shape
"<f8", // data type, here little-endian 64-bit floating point
'C', // memory layout
'/', // chunk identifier separator
xt::xio_binary_config(), // compressor (here, raw binary)
attrs, // attributes
10, // chunk pool size
0. // fill value
);
// write array data
a(2, 1) = 3.;
// read array data
std::cout << a(2, 1) << std::endl;
// prints `3.`
std::cout << a(2, 2) << std::endl;
// prints `0.` (fill value)
std::cout << a.attrs() << std::endl;
// prints `{"answer":42,"question":"life"}`
// create a group
auto g = h.create_group("/tricia/mcmillan", attrs);
// explore the hierarchy
// get children at a point in the hierarchy
std::string children = h.get_children("/").dump();
std::cout << children << std::endl;
// prints `{"arthur":"implicit_group","marvin":"explicit_group","tricia":"implicit_group"}`
// view the whole hierarchy
std::string nodes = h.get_nodes().dump();
std::cout << nodes << std::endl;
// prints `{"arthur":"implicit_group","arthur/dent":"array","tricia":"implicit_group","tricia/mcmillan":"explicit_group"}`
// use cloud storage
// create an anonymous Google Cloud Storage client
gcs::Client client((gcs::ClientOptions(gcs::oauth2::CreateAnonymousCredentials())));
xzarr_gcs_store s1("zarr-demo/v3/test.zr3", client);
// list keys under prefix
auto keys1 = s1.list_prefix("data/root/arthur/dent/");
for (const auto& key: keys1)
{
std::cout << key << std::endl;
}
// prints:
// data/root/arthur/dent/c0/0
// data/root/arthur/dent/c0/1
// data/root/arthur/dent/c1/0
// data/root/arthur/dent/c1/1
// data/root/arthur/dent/c2/0
// data/root/arthur/dent/c2/1
xzarr_gcs_store s2("zarr-demo/v3/test.zr3/meta/root/marvin", client);
// list all keys
auto keys2 = s2.list();
for (const auto& key: keys2)
{
std::cout << key << std::endl;
}
// prints:
// android.array.json
// paranoid.group.jsonLicense
We use a shared copyright model that enables all contributors to maintain the copyright on their contributions.
This software is licensed under the BSD-3-Clause license. See the LICENSE file for details.