xarray zarr s3

Zarr can store arrays in cloud-based object storage such as s3 as chunked, compressed, N-dimensional arrays.

Open a single file:

path = f"s3://{bucket}/dataset.zarr"
s3_store = zarr.storage.FsspecStore.from_url(path)
ds = xr.open_zarr(store=s3_store, consolidated=True, chunks="auto")

Open multiple files as a single dataset in parallel (using dask) and combine along a dimension:

s3 = s3fs.S3FileSystem()
zarr_files = s3.glob(f"{bucket}/*.zarr")
 
ds = zr.open_mfdataset(
	[f"s3://{file}" for file in zarr_files],
	engine="zarr",
	concat_dim="dim_to_concat_along",
	combine="nested",
	parallel=True,
)