The zarr utils command line interface#
This section introduces the usage of the freva-client zarr-utils sub command.
Please see the Installation and configuration section on how to install and
configure the command line interface.
After successful installation you can use the freva-client zarr-utils sub
command
freva-client zarr-utils --help
Converting your data files#
You can convert any data files on the data center to zarr stores and serve them
via http(s). To do so use the convert sub-command:
freva-client zarr-utils convert --help
Results
[1m [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client zarr-utils convert [OPTIONS] {paths}...[0m[1m [0m[1m [0m
[1m [0m
Convert different data sets (files) to a http ready zarr store.
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [31m*[0m paths [1;2;33m<[0m[1;33mstr[0m[1;2;33m>[0m Paths to data files that should be aggregated. [2m│[0m
[2m│[0m [2;31m[required] [0m [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-public[0m [1;33m [0m Make any zarr url [2m│[0m
[2m│[0m public [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-ttl[0m[1;36m-seconds[0m [1;2;33m<[0m[1;33mfloat[0m[1;2;33m>[0m[1;33m [0m Set the expiry time of [2m│[0m
[2m│[0m any public zarr urls [2m│[0m
[2m│[0m [2m[default: 86400.0] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-aggregate[0m [1;2;33m<[0m[1;33mauto[0m[1;2;33m|[0m[1;33mmerge[0m[1;2;33m|[0m[1;33mconcat[0m[1;2;33m>[0m[1;33m [0m How aggregation should [2m│[0m
[2m│[0m be realised (if any) [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-join[0m [1;2;33m<[0m[1;33mouter[0m[1;2;33m|[0m[1;33minner[0m[1;2;33m|[0m[1;33mexact[0m[1;2;33m|[0m[1;33ml[0m How different indexes [2m│[0m
[2m│[0m [1;33meft[0m[1;2;33m|[0m[1;33mright[0m[1;2;33m>[0m[1;33m [0m should be combined for [2m│[0m
[2m│[0m aggregation. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-compat[0m [1;2;33m<[0m[1;33mno_conflicts[0m[1;2;33m|[0m[1;33mequals[0m How to compare [2m│[0m
[2m│[0m [1;2;33m|[0m[1;33moverride[0m[1;2;33m>[0m[1;33m [0m non-concatenated [2m│[0m
[2m│[0m variables for [2m│[0m
[2m│[0m aggregation. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-data[0m[1;36m-vars[0m [1;2;33m<[0m[1;33mminimal[0m[1;2;33m|[0m[1;33mdifferent[0m[1;2;33m|[0m[1;33ma[0m How to combine data [2m│[0m
[2m│[0m [1;33mll[0m[1;2;33m>[0m[1;33m [0m variables for [2m│[0m
[2m│[0m aggregation. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-coords[0m [1;2;33m<[0m[1;33mminimal[0m[1;2;33m|[0m[1;33mdifferent[0m[1;2;33m|[0m[1;33ma[0m How to combine coords [2m│[0m
[2m│[0m [1;33mll[0m[1;2;33m>[0m[1;33m [0m for aggregation. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-dim[0m [1;2;33m<[0m[1;33mstr[0m[1;2;33m>[0m[1;33m [0m Name of the dimension [2m│[0m
[2m│[0m to concatenate along [2m│[0m
[2m│[0m for aggregation. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-group[0m[1;36m-by[0m [1;2;33m<[0m[1;33mstr[0m[1;2;33m>[0m[1;33m [0m If set, forces [2m│[0m
[2m│[0m grouping by a [2m│[0m
[2m│[0m signature key for [2m│[0m
[2m│[0m aggregation. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-access[0m[1;36m-pattern[0m [1;2;33m<[0m[1;33mmap[0m[1;2;33m|[0m[1;33mtime_series[0m[1;2;33m>[0m[1;33m [0m Optimise the chunk [2m│[0m
[2m│[0m sizes for those access [2m│[0m
[2m│[0m pattern. [2m│[0m
[2m│[0m [2m[default: map] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-map[0m[1;36m-primary-chunks…[0m [1;2;33m<[0m[1;33mint[0m[1;2;33m>[0m[1;33m [0m Chunk sizes of the [2m│[0m
[2m│[0m primary dimension. [2m│[0m
[2m│[0m [2m[default: 1] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-chunk[0m[1;36m-size[0m [1;2;33m<[0m[1;33mfloat[0m[1;2;33m>[0m[1;33m [0m Set the target chunk [2m│[0m
[2m│[0m size in megabytes. [2m│[0m
[2m│[0m [2m[default: 16.0] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-reload[0m[1;36m-zarr[0m [1;33m [0m Trigger a zarr [2m│[0m
[2m│[0m data-store reload. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-token[0m[1;36m-file[0m [1;32m-tf[0m [1;2;33m<[0m[1;33mpath[0m[1;2;33m>[0m[1;33m [0m Instead of [2m│[0m
[2m│[0m authenticating via [2m│[0m
[2m│[0m code based [2m│[0m
[2m│[0m authentication flow [2m│[0m
[2m│[0m you can set the path [2m│[0m
[2m│[0m to the json file that [2m│[0m
[2m│[0m contains a `refresh [2m│[0m
[2m│[0m token` containing a [2m│[0m
[2m│[0m refresh_token key. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-json[0m [1;32m-j[0m [1;33m [0m Parse output in json [2m│[0m
[2m│[0m format. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m [1;2;33m<[0m[1;33mstr[0m[1;2;33m>[0m[1;33m [0m Set the hostname of [2m│[0m
[2m│[0m the databrowser, if [2m│[0m
[2m│[0m not set (default) the [2m│[0m
[2m│[0m hostname is read from [2m│[0m
[2m│[0m a config file [2m│[0m
[2m│[0m [1;32m-v[0m [1;2;33m<[0m[1;33mint[0m[1;2;33m>[0m[1;33m [0m Increase verbosity [2m│[0m
[2m│[0m [2m[default: 0] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m [1;32m-V[0m [1;33m [0m Show version an exit [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m [1;33m [0m Show this message and [2m│[0m
[2m│[0m exit. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
The --public flag lets you create zarr stores that can be anonymously acceesed.
The aggregation flags can be used to aggregate multiple source file into one common zarr store.
Warning
Anyone with this link will be able to access the data. For security reasons any public zarr store link has an expiration date after which access isn’t possible anymore.
Checking the conversion status#
If client tools, like xarray, that access the zarr stores file you can use the
status sub-command to get more information about the loading process and
potential issues.
freva-client zarr-utils status --help
Results
[1m [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client zarr-utils status [OPTIONS] {url}[0m[1m [0m[1m [0m
[1m [0m
Get the status of a pre signed zarr store
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [31m*[0m url [1;2;33m<[0m[1;33mstr[0m[1;2;33m>[0m Url of the zarr store to check. [2;31m[required][0m [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-token[0m[1;36m-file[0m [1;32m-tf[0m [1;2;33m<[0m[1;33mpath[0m[1;2;33m>[0m Instead of authenticating via code based [2m│[0m
[2m│[0m authentication flow you can set the path to [2m│[0m
[2m│[0m the json file that contains a `refresh token` [2m│[0m
[2m│[0m containing a refresh_token key. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m [1;2;33m<[0m[1;33mstr[0m[1;2;33m>[0m[1;33m [0m Set the hostname of the databrowser, if not [2m│[0m
[2m│[0m set (default) the hostname is read from a [2m│[0m
[2m│[0m config file [2m│[0m
[2m│[0m [1;32m-v[0m [1;2;33m<[0m[1;33mint[0m[1;2;33m>[0m[1;33m [0m Increase verbosity [2m[default: 0][0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m [1;32m-V[0m [1;33m [0m Show version an exit [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m [1;33m [0m Show this message and exit. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m