The databrowser command line interface#
This section introduces the usage of the freva-client databrowser sub command.
Please see the Installation and configuration section on how to install and
configure the command line interface.
After successful installation you can use the freva-client databrowser sub
command
freva-client databrowser --help
Searching for data locations#
Getting the locations of the data is probably the most common use case of the
databrowser application. You can search for data locations by applying the
data-search sub-command:
freva-client databrowser data-search --help
Results
[1m [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser data-search [OPTIONS] [SEARCH_KEYS]...[0m[1m [0m[1m [0m
[1m [0m
Search the databrowser for datasets.
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m search_keys [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m Refine your data search with this [2m│[0m
[2m│[0m `key=value` pair search parameters. The [2m│[0m
[2m│[0m parameters could be, depending on the [2m│[0m
[2m│[0m DRS standard, flavour product, project [2m│[0m
[2m│[0m model etc. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m [1;33mTEXT [0m If you are not sure [2m│[0m
[2m│[0m about the correct search [2m│[0m
[2m│[0m key's you can use the [2m│[0m
[2m│[0m ``[1;36m-[0m[1;36m-facet[0m`` flag to [2m│[0m
[2m│[0m search of any matching [2m│[0m
[2m│[0m entries. For example [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m 'era5' would [2m│[0m
[2m│[0m allow you to search for [2m│[0m
[2m│[0m any entries containing [2m│[0m
[2m│[0m era5, regardless of [2m│[0m
[2m│[0m project, product etc. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-uniq[0m[1;36m-key[0m [1;32m-u[0m [1;2;33m[[0m[1;33mfile[0m[1;2;33m|[0m[1;33muri[0m[1;2;33m][0m[1;33m [0m The type of search [2m│[0m
[2m│[0m result, which can be [2m│[0m
[2m│[0m either “file” or “uri”. [2m│[0m
[2m│[0m This parameter [2m│[0m
[2m│[0m determines whether the [2m│[0m
[2m│[0m search will be based on [2m│[0m
[2m│[0m file paths or Uniform [2m│[0m
[2m│[0m Resource Identifiers [2m│[0m
[2m│[0m [2m[default: file] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-flavour[0m [1;33mTEXT [0m The Data Reference [2m│[0m
[2m│[0m Syntax (DRS) standard [2m│[0m
[2m│[0m specifying the type of [2m│[0m
[2m│[0m climate datasets to [2m│[0m
[2m│[0m query. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m [1;32m-ts[0m [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m Operator that specifies [2m│[0m
[2m│[0m how the time period is [2m│[0m
[2m│[0m selected. Choose from [2m│[0m
[2m│[0m flexible (default), [2m│[0m
[2m│[0m strict or file. [2m│[0m
[2m│[0m ``strict`` returns only [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m the *entire* time period [2m│[0m
[2m│[0m covered. The time search [2m│[0m
[2m│[0m ``2000 to 2012`` will [2m│[0m
[2m│[0m not select files [2m│[0m
[2m│[0m containing data from [2m│[0m
[2m│[0m 2010 to 2020 with the [2m│[0m
[2m│[0m ``strict`` method. [2m│[0m
[2m│[0m ``flexible`` will select [2m│[0m
[2m│[0m those files as [2m│[0m
[2m│[0m ``flexible`` returns [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m start or end period [2m│[0m
[2m│[0m covered. ``file`` will [2m│[0m
[2m│[0m only return files where [2m│[0m
[2m│[0m the entire time period [2m│[0m
[2m│[0m is contained within *one [2m│[0m
[2m│[0m single* file. [2m│[0m
[2m│[0m [2m[default: flexible] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-zarr[0m [1;33m [0m Create zarr stream [2m│[0m
[2m│[0m files. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-public[0m [1;33m [0m Make any zarr url public [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-ttl[0m[1;36m-seconds[0m [1;33mINTEGER [0m Set the expiry time of [2m│[0m
[2m│[0m any public zarr urls [2m│[0m
[2m│[0m [2m[default: 86400] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-token[0m[1;36m-file[0m [1;32m-tf[0m [1;33mPATH [0m Instead of [2m│[0m
[2m│[0m authenticating via code [2m│[0m
[2m│[0m based authentication [2m│[0m
[2m│[0m flow you can set the [2m│[0m
[2m│[0m path to the json file [2m│[0m
[2m│[0m that contains a `refresh [2m│[0m
[2m│[0m token` containing a [2m│[0m
[2m│[0m refresh_token key. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m [1;32m-t[0m [1;33mTEXT [0m Special search facet to [2m│[0m
[2m│[0m refine/subset search [2m│[0m
[2m│[0m results by time. This [2m│[0m
[2m│[0m can be a string [2m│[0m
[2m│[0m representation of a time [2m│[0m
[2m│[0m range or a single time [2m│[0m
[2m│[0m step. The time steps [2m│[0m
[2m│[0m have to follow ISO-8601. [2m│[0m
[2m│[0m Valid strings are [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M`` to [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M`` for [2m│[0m
[2m│[0m time ranges and [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M``. [2m│[0m
[2m│[0m **Note**: You don't have [2m│[0m
[2m│[0m to give the full string [2m│[0m
[2m│[0m format to subset time [2m│[0m
[2m│[0m steps ``%Y``, ``%Y-%m`` [2m│[0m
[2m│[0m etc are also valid. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-bbox[0m [1;32m-b[0m [1;2;33m<[0m[1;33mFLOAT FLOAT FLOAT [0m Special search facet to [2m│[0m
[2m│[0m [1;33mFLOAT>... [0m refine/subset search [2m│[0m
[2m│[0m results by spatial [2m│[0m
[2m│[0m extent. This can be a [2m│[0m
[2m│[0m string representation of [2m│[0m
[2m│[0m a bounding box. The [2m│[0m
[2m│[0m bounding box has to [2m│[0m
[2m│[0m follow the format [2m│[0m
[2m│[0m ``min_lon max_lon [2m│[0m
[2m│[0m min_lat max_lat``. Valid [2m│[0m
[2m│[0m strings are ``[1;32m-10[0m 10 [1;32m-10[0m [2m│[0m
[2m│[0m 10`` to ``0 5 0 5``. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-bbox[0m[1;36m-select[0m [1;32m-bs[0m [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m Operator that specifies [2m│[0m
[2m│[0m how the spatial extent [2m│[0m
[2m│[0m is selected. Choose from [2m│[0m
[2m│[0m flexible (default), [2m│[0m
[2m│[0m strict or file. [2m│[0m
[2m│[0m ``strict`` returns only [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m the *entire* spatial [2m│[0m
[2m│[0m extent covered. The bbox [2m│[0m
[2m│[0m search ``[1;32m-10[0m 10 [1;32m-10[0m 10`` [2m│[0m
[2m│[0m will not select files [2m│[0m
[2m│[0m containing data from 0 5 [2m│[0m
[2m│[0m 0 5 with the ``strict`` [2m│[0m
[2m│[0m method. ``flexible`` [2m│[0m
[2m│[0m will select those files [2m│[0m
[2m│[0m as ``flexible`` returns [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m any part of the extent [2m│[0m
[2m│[0m covered. ``file`` will [2m│[0m
[2m│[0m only return files where [2m│[0m
[2m│[0m the entire spatial [2m│[0m
[2m│[0m extent is contained [2m│[0m
[2m│[0m within *one single* [2m│[0m
[2m│[0m file. [2m│[0m
[2m│[0m [2m[default: flexible] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-json[0m [1;32m-j[0m [1;33m [0m Parse output in json [2m│[0m
[2m│[0m format. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m [1;33mTEXT [0m Set the hostname of the [2m│[0m
[2m│[0m databrowser, if not set [2m│[0m
[2m│[0m (default) the hostname [2m│[0m
[2m│[0m is read from a config [2m│[0m
[2m│[0m file [2m│[0m
[2m│[0m [1;32m-v[0m [1;33mINTEGER [0m Increase verbosity [2m│[0m
[2m│[0m [2m[default: 0] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m [1;33m [0m Select all versions and [2m│[0m
[2m│[0m not just the latest [2m│[0m
[2m│[0m version (default). [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m [1;32m-V[0m [1;33m [0m Show version an exit [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m [1;33m [0m Show this message and [2m│[0m
[2m│[0m exit. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
The command expects a list of key=value pairs. The order of the pairs doesn’t really matter. Most important is that you don’t need to split the search according to the type of data you are searching for. You can search for files within observations, reanalysis and model data at the same time. Also important is that all queries are case insensitive. You can also search for attributes themselves instead of file paths. For example you can search for the list of variables available that satisfies a certain constraint (e.g. sampled 6hr, from a certain model, etc).
freva-client databrowser data-search project=observations variable=pr model=cp*
There are many more options for defining a value for a given key:
Attribute syntax |
Meaning |
|---|---|
|
Search for files containing exactly that attribute |
|
Search for files containing a value for attribute that starts with the prefix val |
|
Search for files containing a value for attribute that ends with the suffix lue |
|
Search for files containing a value for attribute that has alu somewhere |
|
Search for files containing a value for attribute that matches the given regular expression (yes! you might use any regular expression to find what you want.) |
OR:
|
Search for files containing either value1 OR value2 for the given attribute (note that’s the same attribute twice!) |
|
Search for files containing value1 for attribute1 AND value2 for attribute2 |
|
Search for files NOT containing value |
|
Search for files containing neither value1 nor value2 |
Note
When using * remember that your shell might give it a different meaning (normally it will try to match files with that name) to turn that off you can use backslash (key=*) or use quotes (key=’*’).
Searching multi-versioned datasets#
In datasets with multiple versions only the latest version (i.e. highest
version number) is returned by default. Querying a specific version from a
multi versioned datasets requires the multi-version flag in combination with
the version special attribute:
freva-client databrowser data-search dataset=cmip6-fs model=access-cm2 --multi-version version=v20191108
If no particular version is requested, all versions will be returned.
Streaming files via zarr#
Instead of getting the file locations on disk or tape, you can instruct the
system to register zarr streams. Which means that instead of opening the
data directly you can open it via zarr from anywhere. To do so simply add
the --zarr flag.
Note
Before you can use the --zarr flag you will have
to create an access token and use that token to log on to the system
see also the Authentication & Authorization chapter for more details on token creation.
freva-client auth > .token.json
chmod 600 .token.json
freva-client databrowser data-search dataset=cmip6-fs --zarr --token-file .token.json
Special cases: Searching for times#
For example you want to know how many files we have between a certain time range To do so you can use the time search key to subset time steps and whole time ranges:
freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10'
The default method for selecting time periods is flexible, which means
all files are selected that cover at least start or end date. The
strict method implies that the entire search time period has to be
covered by the files. Using the strict method in the example above would
only yield on file because the first file contains time steps prior to the
start of the time period:
freva-client databrowser data-search project=observations -t '2016-09-02T22:15 to 2016-10' -ts strict
Giving single time steps is also possible:
freva-client databrowser data-search project=observations -t 2016-09-02T22:10
Note
The time format has to follow the
ISO-8601 standard. Time ranges
are indicated by the to keyword such as 2000 to 2100 or
2000-01 to 2100-12 and alike. Single time steps are given without the
to keyword.
Creating intake-esm catalouges#
The intake-catalogue sub command allows you to create an
intake-esm catalogue <https://intake-esm.readthedocs.io/en/stable/>_ from
the current search. This can be useful to share the catalogue with others
or merge datasets.
freva-client databrowser intake-catalogue --help
Results
[1m [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser intake-catalogue [OPTIONS] [SEARCH_KEYS]...[0m[1m [0m[1m [0m
[1m [0m
Create an intake catalogue from the search.
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m search_keys [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m Refine your data search with this [2m│[0m
[2m│[0m `key=value` pair search parameters. The [2m│[0m
[2m│[0m parameters could be, depending on the [2m│[0m
[2m│[0m DRS standard, flavour product, project [2m│[0m
[2m│[0m model etc. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m [1;33mTEXT [0m If you are not sure [2m│[0m
[2m│[0m about the correct search [2m│[0m
[2m│[0m key's you can use the [2m│[0m
[2m│[0m ``[1;36m-[0m[1;36m-facet[0m`` flag to [2m│[0m
[2m│[0m search of any matching [2m│[0m
[2m│[0m entries. For example [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m 'era5' would [2m│[0m
[2m│[0m allow you to search for [2m│[0m
[2m│[0m any entries containing [2m│[0m
[2m│[0m era5, regardless of [2m│[0m
[2m│[0m project, product etc. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-uniq[0m[1;36m-key[0m [1;32m-u[0m [1;2;33m[[0m[1;33mfile[0m[1;2;33m|[0m[1;33muri[0m[1;2;33m][0m[1;33m [0m The type of search [2m│[0m
[2m│[0m result, which can be [2m│[0m
[2m│[0m either “file” or “uri”. [2m│[0m
[2m│[0m This parameter [2m│[0m
[2m│[0m determines whether the [2m│[0m
[2m│[0m search will be based on [2m│[0m
[2m│[0m file paths or Uniform [2m│[0m
[2m│[0m Resource Identifiers [2m│[0m
[2m│[0m [2m[default: file] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-flavour[0m [1;33mTEXT [0m The Data Reference [2m│[0m
[2m│[0m Syntax (DRS) standard [2m│[0m
[2m│[0m specifying the type of [2m│[0m
[2m│[0m climate datasets to [2m│[0m
[2m│[0m query. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m [1;32m-ts[0m [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m Operator that specifies [2m│[0m
[2m│[0m how the time period is [2m│[0m
[2m│[0m selected. Choose from [2m│[0m
[2m│[0m flexible (default), [2m│[0m
[2m│[0m strict or file. [2m│[0m
[2m│[0m ``strict`` returns only [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m the *entire* time period [2m│[0m
[2m│[0m covered. The time search [2m│[0m
[2m│[0m ``2000 to 2012`` will [2m│[0m
[2m│[0m not select files [2m│[0m
[2m│[0m containing data from [2m│[0m
[2m│[0m 2010 to 2020 with the [2m│[0m
[2m│[0m ``strict`` method. [2m│[0m
[2m│[0m ``flexible`` will select [2m│[0m
[2m│[0m those files as [2m│[0m
[2m│[0m ``flexible`` returns [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m start or end period [2m│[0m
[2m│[0m covered. ``file`` will [2m│[0m
[2m│[0m only return files where [2m│[0m
[2m│[0m the entire time period [2m│[0m
[2m│[0m is contained within *one [2m│[0m
[2m│[0m single* file. [2m│[0m
[2m│[0m [2m[default: flexible] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m [1;32m-t[0m [1;33mTEXT [0m Special search facet to [2m│[0m
[2m│[0m refine/subset search [2m│[0m
[2m│[0m results by time. This [2m│[0m
[2m│[0m can be a string [2m│[0m
[2m│[0m representation of a time [2m│[0m
[2m│[0m range or a single time [2m│[0m
[2m│[0m step. The time steps [2m│[0m
[2m│[0m have to follow ISO-8601. [2m│[0m
[2m│[0m Valid strings are [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M`` to [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M`` for [2m│[0m
[2m│[0m time ranges and [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M``. [2m│[0m
[2m│[0m **Note**: You don't have [2m│[0m
[2m│[0m to give the full string [2m│[0m
[2m│[0m format to subset time [2m│[0m
[2m│[0m steps ``%Y``, ``%Y-%m`` [2m│[0m
[2m│[0m etc are also valid. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-bbox[0m [1;32m-b[0m [1;2;33m<[0m[1;33mFLOAT FLOAT FLOAT [0m Special search facet to [2m│[0m
[2m│[0m [1;33mFLOAT>... [0m refine/subset search [2m│[0m
[2m│[0m results by spatial [2m│[0m
[2m│[0m extent. This can be a [2m│[0m
[2m│[0m string representation of [2m│[0m
[2m│[0m a bounding box. The [2m│[0m
[2m│[0m bounding box has to [2m│[0m
[2m│[0m follow the format [2m│[0m
[2m│[0m ``min_lon max_lon [2m│[0m
[2m│[0m min_lat max_lat``. Valid [2m│[0m
[2m│[0m strings are ``[1;32m-10[0m 10 [1;32m-10[0m [2m│[0m
[2m│[0m 10`` to ``0 5 0 5``. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-bbox[0m[1;36m-select[0m [1;32m-bs[0m [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m Operator that specifies [2m│[0m
[2m│[0m how the spatial extent [2m│[0m
[2m│[0m is selected. Choose from [2m│[0m
[2m│[0m flexible (default), [2m│[0m
[2m│[0m strict or file. [2m│[0m
[2m│[0m ``strict`` returns only [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m the *entire* spatial [2m│[0m
[2m│[0m extent covered. The bbox [2m│[0m
[2m│[0m search ``[1;32m-10[0m 10 [1;32m-10[0m 10`` [2m│[0m
[2m│[0m will not select files [2m│[0m
[2m│[0m containing data from 0 5 [2m│[0m
[2m│[0m 0 5 with the ``strict`` [2m│[0m
[2m│[0m method. ``flexible`` [2m│[0m
[2m│[0m will select those files [2m│[0m
[2m│[0m as ``flexible`` returns [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m any part of the extent [2m│[0m
[2m│[0m covered. ``file`` will [2m│[0m
[2m│[0m only return files where [2m│[0m
[2m│[0m the entire spatial [2m│[0m
[2m│[0m extent is contained [2m│[0m
[2m│[0m within *one single* [2m│[0m
[2m│[0m file. [2m│[0m
[2m│[0m [2m[default: flexible] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-zarr[0m [1;33m [0m Create zarr stream [2m│[0m
[2m│[0m files, as catalogue [2m│[0m
[2m│[0m targets. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-public[0m [1;33m [0m Make any zarr url public [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-ttl[0m[1;36m-seconds[0m [1;33mINTEGER [0m Set the expiry time of [2m│[0m
[2m│[0m any public zarr urls [2m│[0m
[2m│[0m [2m[default: 86400] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-token[0m[1;36m-file[0m [1;32m-tf[0m [1;33mPATH [0m Instead of [2m│[0m
[2m│[0m authenticating via code [2m│[0m
[2m│[0m based authentication [2m│[0m
[2m│[0m flow you can set the [2m│[0m
[2m│[0m path to the json file [2m│[0m
[2m│[0m that contains a `refresh [2m│[0m
[2m│[0m token` containing a [2m│[0m
[2m│[0m refresh_token key. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-filename[0m [1;32m-f[0m [1;33mPATH [0m Path to the file where [2m│[0m
[2m│[0m the catalogue, should be [2m│[0m
[2m│[0m written to. if None [2m│[0m
[2m│[0m given (default) the [2m│[0m
[2m│[0m catalogue is parsed to [2m│[0m
[2m│[0m stdout. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m [1;33mTEXT [0m Set the hostname of the [2m│[0m
[2m│[0m databrowser, if not set [2m│[0m
[2m│[0m (default) the hostname [2m│[0m
[2m│[0m is read from a config [2m│[0m
[2m│[0m file [2m│[0m
[2m│[0m [1;32m-v[0m [1;33mINTEGER [0m Increase verbosity [2m│[0m
[2m│[0m [2m[default: 0] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m [1;33m [0m Select all versions and [2m│[0m
[2m│[0m not just the latest [2m│[0m
[2m│[0m version (default). [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m [1;32m-V[0m [1;33m [0m Show version an exit [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m [1;33m [0m Show this message and [2m│[0m
[2m│[0m exit. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
You can either set the --filename flag to save the catalogue to a .json
file or pipe the catalogue to stdout (default). Just like for the data-search
sub command you can instruct the system to create zarr file streams to access
the data via zarr.
Creating STAC Catalogue#
The stac-catalogue sub command allows you to create a static
SpatioTemporal Asset Catalog (STAC) <https://stacspec.org/en/about/stac-spec/>_
from the current search. This can be useful for creating, sharing and using
standardized geospatial data catalogs and enabling interoperability between
different data systems.
freva-client databrowser stac-catalogue --help
Results
[1m [0m
[1m [0m[1;33mUsage: [0m[1mfreva-client databrowser stac-catalogue [OPTIONS] [SEARCH_KEYS]...[0m[1m [0m[1m [0m
[1m [0m
Create a static STAC catalogue from the search.
[2m╭─[0m[2m Arguments [0m[2m─────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m search_keys [1;2;33m[[0m[1;33mSEARCH_KEYS]...[0m Refine your data search with this [2m│[0m
[2m│[0m `key=value` pair search parameters. The [2m│[0m
[2m│[0m parameters could be, depending on the [2m│[0m
[2m│[0m DRS standard, flavour product, project [2m│[0m
[2m│[0m model etc. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
[2m╭─[0m[2m Options [0m[2m───────────────────────────────────────────────────────────────────[0m[2m─╮[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m [1;33mTEXT [0m If you are not sure [2m│[0m
[2m│[0m about the correct search [2m│[0m
[2m│[0m key's you can use the [2m│[0m
[2m│[0m ``[1;36m-[0m[1;36m-facet[0m`` flag to [2m│[0m
[2m│[0m search of any matching [2m│[0m
[2m│[0m entries. For example [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-facet[0m 'era5' would [2m│[0m
[2m│[0m allow you to search for [2m│[0m
[2m│[0m any entries containing [2m│[0m
[2m│[0m era5, regardless of [2m│[0m
[2m│[0m project, product etc. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-uniq[0m[1;36m-key[0m [1;32m-u[0m [1;2;33m[[0m[1;33mfile[0m[1;2;33m|[0m[1;33muri[0m[1;2;33m][0m[1;33m [0m The type of search [2m│[0m
[2m│[0m result, which can be [2m│[0m
[2m│[0m either “file” or “uri”. [2m│[0m
[2m│[0m This parameter [2m│[0m
[2m│[0m determines whether the [2m│[0m
[2m│[0m search will be based on [2m│[0m
[2m│[0m file paths or Uniform [2m│[0m
[2m│[0m Resource Identifiers [2m│[0m
[2m│[0m [2m[default: file] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-flavour[0m [1;33mTEXT [0m The Data Reference [2m│[0m
[2m│[0m Syntax (DRS) standard [2m│[0m
[2m│[0m specifying the type of [2m│[0m
[2m│[0m climate datasets to [2m│[0m
[2m│[0m query. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m[1;36m-select[0m [1;32m-ts[0m [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m Operator that specifies [2m│[0m
[2m│[0m how the time period is [2m│[0m
[2m│[0m selected. Choose from [2m│[0m
[2m│[0m flexible (default), [2m│[0m
[2m│[0m strict or file. [2m│[0m
[2m│[0m ``strict`` returns only [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m the *entire* time period [2m│[0m
[2m│[0m covered. The time search [2m│[0m
[2m│[0m ``2000 to 2012`` will [2m│[0m
[2m│[0m not select files [2m│[0m
[2m│[0m containing data from [2m│[0m
[2m│[0m 2010 to 2020 with the [2m│[0m
[2m│[0m ``strict`` method. [2m│[0m
[2m│[0m ``flexible`` will select [2m│[0m
[2m│[0m those files as [2m│[0m
[2m│[0m ``flexible`` returns [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m start or end period [2m│[0m
[2m│[0m covered. ``file`` will [2m│[0m
[2m│[0m only return files where [2m│[0m
[2m│[0m the entire time period [2m│[0m
[2m│[0m is contained within *one [2m│[0m
[2m│[0m single* file. [2m│[0m
[2m│[0m [2m[default: flexible] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-time[0m [1;32m-t[0m [1;33mTEXT [0m Special search facet to [2m│[0m
[2m│[0m refine/subset search [2m│[0m
[2m│[0m results by time. This [2m│[0m
[2m│[0m can be a string [2m│[0m
[2m│[0m representation of a time [2m│[0m
[2m│[0m range or a single time [2m│[0m
[2m│[0m step. The time steps [2m│[0m
[2m│[0m have to follow ISO-8601. [2m│[0m
[2m│[0m Valid strings are [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M`` to [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M`` for [2m│[0m
[2m│[0m time ranges and [2m│[0m
[2m│[0m ``%Y-%m-%dT%H:%M``. [2m│[0m
[2m│[0m **Note**: You don't have [2m│[0m
[2m│[0m to give the full string [2m│[0m
[2m│[0m format to subset time [2m│[0m
[2m│[0m steps ``%Y``, ``%Y-%m`` [2m│[0m
[2m│[0m etc are also valid. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-bbox[0m [1;32m-b[0m [1;2;33m<[0m[1;33mFLOAT FLOAT FLOAT [0m Special search facet to [2m│[0m
[2m│[0m [1;33mFLOAT>... [0m refine/subset search [2m│[0m
[2m│[0m results by spatial [2m│[0m
[2m│[0m extent. This can be a [2m│[0m
[2m│[0m string representation of [2m│[0m
[2m│[0m a bounding box. The [2m│[0m
[2m│[0m bounding box has to [2m│[0m
[2m│[0m follow the format [2m│[0m
[2m│[0m ``min_lon max_lon [2m│[0m
[2m│[0m min_lat max_lat``. Valid [2m│[0m
[2m│[0m strings are ``[1;32m-10[0m 10 [1;32m-10[0m [2m│[0m
[2m│[0m 10`` to ``0 5 0 5``. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-bbox[0m[1;36m-select[0m [1;32m-bs[0m [1;2;33m[[0m[1;33mstrict[0m[1;2;33m|[0m[1;33mflexible[0m[1;2;33m|[0m[1;33mfile[0m[1;2;33m][0m[1;33m [0m Operator that specifies [2m│[0m
[2m│[0m how the spatial extent [2m│[0m
[2m│[0m is selected. Choose from [2m│[0m
[2m│[0m flexible (default), [2m│[0m
[2m│[0m strict or file. [2m│[0m
[2m│[0m ``strict`` returns only [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m the *entire* spatial [2m│[0m
[2m│[0m extent covered. The bbox [2m│[0m
[2m│[0m search ``[1;32m-10[0m 10 [1;32m-10[0m 10`` [2m│[0m
[2m│[0m will not select files [2m│[0m
[2m│[0m containing data from 0 5 [2m│[0m
[2m│[0m 0 5 with the ``strict`` [2m│[0m
[2m│[0m method. ``flexible`` [2m│[0m
[2m│[0m will select those files [2m│[0m
[2m│[0m as ``flexible`` returns [2m│[0m
[2m│[0m those files that have [2m│[0m
[2m│[0m any part of the extent [2m│[0m
[2m│[0m covered. ``file`` will [2m│[0m
[2m│[0m only return files where [2m│[0m
[2m│[0m the entire spatial [2m│[0m
[2m│[0m extent is contained [2m│[0m
[2m│[0m within *one single* [2m│[0m
[2m│[0m file. [2m│[0m
[2m│[0m [2m[default: flexible] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-host[0m [1;33mTEXT [0m Set the hostname of the [2m│[0m
[2m│[0m databrowser, if not set [2m│[0m
[2m│[0m (default) the hostname [2m│[0m
[2m│[0m is read from a config [2m│[0m
[2m│[0m file [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-token[0m[1;36m-file[0m [1;32m-tf[0m [1;33mPATH [0m Instead of [2m│[0m
[2m│[0m authenticating via code [2m│[0m
[2m│[0m based authentication [2m│[0m
[2m│[0m flow you can set the [2m│[0m
[2m│[0m path to the json file [2m│[0m
[2m│[0m that contains a `refresh [2m│[0m
[2m│[0m token` containing a [2m│[0m
[2m│[0m refresh_token key. [2m│[0m
[2m│[0m [1;32m-v[0m [1;33mINTEGER [0m Increase verbosity [2m│[0m
[2m│[0m [2m[default: 0] [0m [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-multi[0m[1;36m-version[0m [1;33m [0m Select all versions and [2m│[0m
[2m│[0m not just the latest [2m│[0m
[2m│[0m version (default). [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-version[0m [1;32m-V[0m [1;33m [0m Show version an exit [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-filename[0m [1;32m-o[0m [1;33mPATH [0m Path to the file where [2m│[0m
[2m│[0m the static STAC [2m│[0m
[2m│[0m catalogue, should be [2m│[0m
[2m│[0m written to. If you don't [2m│[0m
[2m│[0m specify or the path does [2m│[0m
[2m│[0m not exist, the file will [2m│[0m
[2m│[0m be created in the [2m│[0m
[2m│[0m current working [2m│[0m
[2m│[0m directory. [2m│[0m
[2m│[0m [1;36m-[0m[1;36m-help[0m [1;33m [0m Show this message and [2m│[0m
[2m│[0m exit. [2m│[0m
[2m╰──────────────────────────────────────────────────────────────────────────────╯[0m
To get an static STAC catalogue you can use the following command:
freva-client databrowser stac-catalogue --filename /path/to/output
and if the specified filename directory doesn’t specify or not existed or not provided, the STAC catalogue will be saved in the current directory. It can be only a directory or a fully qualified filename.
The STAC Catalogue provides multiple ways to access and interact with the data:
Access your climate data through the intake-esm data catalog specification
Access search results as Zarr files, available as STAC Assets at both collection and item levels
Browse and explore your search results directly through the Freva DataBrowser web interface
Each of these access methods is encoded as STAC Assets, making them easily discoverable and accessible through any STAC-compatible tool.
Query the number of occurrences#
In some cases it might be useful to know how many files are found in the
databrowser for certain search constraints. In such cases you can use the
data-count sub command to count the number of found files instead of getting
the files themselves.
freva-client databrowser data-count --help
By default the data-count sub command will display the total number of items
matching your search query. For example:
freva-client databrowser data-count project=observations
If you want to group the number of occurrences by search categories (facets)
use the -d or --detail flag:
freva-client databrowser data-count -d project=observations
Retrieving the available metadata#
Sometime it might be useful to know about possible values that search attributes.
For this you use the metadata-search sub command:
freva-client databrowser metadata-search --help
Just like with any other databrowser command you can apply different search constraints when acquiring metadata
freva-client databrowser metadata-search project=observations
By default the command will display only the most commonly used metadata
keys. To retrieve all metadata you can use the -e or --extended-search
flag.
freva-client databrowser metadata-search -e project=observations
Sometimes you don’t exactly know the exact names of the search keys and
want retrieve all file objects that match a certain category. For example
for getting all ocean reanalysis datasets you can apply the --facet flag:
freva-client databrowser metadata-search -e realm=ocean --facet 'rean*'
Filtering metadata results#
You can filter the metadata results using standard command-line tools:
Using grep to filter pandas output:
freva-client databrowser metadata-search | grep -E '(project|realm)'
Using jq to filter JSON output:
freva-client databrowser metadata-search --json | jq 'with_entries(select(.key | IN("experiment", "realm")))'
These methods allow you to focus on specific metadata categories and make the output more targeted and easier to process.
Expert tip: Getting metadata for certain files#
In some cases it might be useful to retrieve metadata for certain file or object store locations. For example if we wanted to retrieve the metadata of those files on tape:
freva-client databrowser metadata-search -e file="/arch/*"
Parsing the command output#
You might have already noticed that each command has a --json flag.
Enabling this flag lets you parse the output of each command to JSON.
Following on from the example above we can parse the output of the reverse search to the command line json processor jq:
freva-client databrowser metadata-search -e file="/arch/*" --json
By using the pipe operator | the JSON output of the freva-client
commands can be piped and processed by jq:
freva-client databrowser metadata-search -e file="/arch/*" --json | jq -r .ensemble[0]
The above example will select only the first entry of the ensembles that are associated with files on the tape archive.
Adding and Deleting User Data#
You can manage your personal datasets within the databrowser by adding or deleting user-specific data. This functionality allows you to include your own data files into the databrowser, making them accessible for analysis and retrieval alongside other datasets.
Before using the user-data commands, you need to create an access token and authenticate with the system. Please refer to the Authentication & Authorization chapter for more details on token creation and authentication.
Adding User Data#
To add your data to the databrowser, use the user-data add command. You’ll need to provide the paths to your data files, and any metadata you’d like to associate with your data.
freva-client databrowser user-data add \
--path freva-rest/src/freva_rest/databrowser_api/mock/ \
--facet project=cordex \
--facet experiment=rcp85 \
--facet model=mpi-m-mpi-esm-lr-clmcom-cclm4-8-17-v1 \
--facet variable=tas \
--token-file ~/.access_token
This command adds the specified data files to the databrowser and tags them with the provided metadata. These search filters help in indexing and searching your data within the system.
Deleting User Data#
If you need to remove your data from the databrowser, use the user-data delete command. Provide your the search keys (facets) that identify the user data you wish to delete.
freva-client databrowser user-data delete \
--search-key project=cordex \
--search-key experiment=rcp85 \
--token-file ~/.access_token
This command deletes all data entries that match the specified search keys from the databrowser.
Managing Custom Flavours#
You can create and manage custom Data Reference Syntax (DRS) flavours that define how search facets are mapped to different data standards. This allows you to create personalized search interfaces that match your specific data organization needs.
Before using the flavour commands for adding or deleting flavours, you need to create an access token and authenticate with the system. Please refer to the Authentication & Authorization chapter for more details on token creation and authentication.
Adding Custom Flavours#
To create a new custom flavour, use the flavour add command. You’ll need to provide a name for your flavour and specify how search facets should be mapped.
freva-client databrowser flavour add my-custom-flavour \
--map project=activity_id \
--map model=source_id \
--map experiment=experiment_id
This command creates a personal flavour named “my-custom-flavour” that maps the standard search keys to different attribute names.
To create a global flavour that all users can access (requires admin privileges):
freva-client databrowser flavour add global-flavour \
--map project=activity_id \
--map model=source_id \
--global
Updating Custom Flavours#
To update an existing custom flavour, use the flavour update command. This allows you to modify the facet mappings without recreating the entire flavour. You can update specific keys while keeping others unchanged.
freva-client databrowser flavour update my-custom-flavour \
--map experiment=new_experiment_id \
--map variable=var_name
This command updates only the specified mappings in “my-custom-flavour”, leaving other mappings intact.
You can also rename a flavour while updating it:
freva-client databrowser flavour update my-custom-flavour \
--new-name my-updated-flavour \
--map project=new_project_id
To update a global flavour (requires admin privileges):
freva-client databrowser flavour update global-flavour \
--map model=updated_source_id \
--global
Note
Regular users can only update their own personal flavours. Admin users can update both personal and global flavours. If you only want to rename a flavour, you can use --new-name without providing any --map options.
Listing Available Flavours#
To see all custom flavours available to you, use the flavour list command:
freva-client databrowser flavour list
Note
The flavour list command does not require authentication and will show global flavours available to all users.
This will display both personal flavours you’ve created and global flavours available to all users.
You can also output the results in JSON format:
freva-client databrowser flavour list --json
Deleting Custom Flavours#
To remove a custom flavour you no longer need, use the flavour delete command:
freva-client databrowser flavour delete my-custom-flavour
This command removes the specified flavour from the system. You can only delete flavours that you own, unless you are an admin user who can delete global flavours.