Skip to content

zarr_local_storage

Classes:

Name Description
ZarrLocalStorage

Handles data storage and retrieval for zarr archives on a local filesystem.

Classes#

ZarrLocalStorage #

Bases: FileSystem

Handles data storage and retrieval for zarr archives on a local filesystem.

Zarr is a special format that writes chunked data to a number of files underneath a given directory. This distribution of data into chunks and distinct files makes zarr an extremely well-suited format for quickly storing and retrieving large quantities of data.

Classes:

Name Description
Parameters

Methods:

Name Description
last_modified
modified_since

Attributes:

Name Type Description
handler ZarrHandler

The ZarrHandler class that should be used to handle data I/O within the storage
parameters Parameters

File-system specific parameters, such as the root path to where the Z arr

Attributes#

handler class-attribute instance-attribute #
handler: ZarrHandler = Field(default_factory=ZarrHandler)

The ZarrHandler class that should be used to handle data I/O within the storage API.

parameters class-attribute instance-attribute #
parameters: Parameters = Field(default_factory=Parameters)

File-system specific parameters, such as the root path to where the Z arr archives should be saved, or additional keyword arguments to specific functions used by the storage API. See the Parameters class for more details.

Classes#

Parameters #

Bases: Parameters

Attributes:

Name Type Description
data_filename_template str

Template string to use for data filenames.
data_storage_path Path

The directory structure under storage_root where ancillary files are saved.
Attributes#
data_filename_template class-attribute instance-attribute # 
data_filename_template: str = '{datastream}.{extension}'

Template string to use for data filenames.

Allows substitution of the following parameters using curly braces '{}':

  • ext: the file extension from the storage data handler
  • datastream from the dataset's global attributes
  • location_id from the dataset's global attributes
  • data_level from the dataset's global attributes
  • Any other global attribute that has a string or integer data type.
data_storage_path class-attribute instance-attribute # 
data_storage_path: Path = Path('data/{location_id}')

The directory structure under storage_root where ancillary files are saved.

Allows substitution of the following parameters using curly braces '{}':

  • storage_root: the value from the storage_root parameter.
  • datastream: the datastream as defined in the dataset config file.
  • location_id: the location_id as defined in the dataset config file.
  • data_level: the data_level as defined in the dataset config file.
  • year: the year of the first timestamp in the file.
  • month: the month of the first timestamp in the file.
  • day: the day of the first timestamp in the file.
  • extension: the file extension used by the output file writer.

Functions#

last_modified #
last_modified(datastream: str) -> datetime | None
Source code in tsdat/io/storage/zarr_local_storage.py 
59
60
61
def last_modified(self, datastream: str) -> datetime | None:
    logger.warning("ZarrLocalStorage does not support last_modified()")
    return None
modified_since #
modified_since(
    datastream: str, last_modified: datetime
) -> List[datetime]
Source code in tsdat/io/storage/zarr_local_storage.py 
63
64
65
66
67
def modified_since(
    self, datastream: str, last_modified: datetime
) -> List[datetime]:
    logger.warning("ZarrLocalStorage does not support modified_since()")
    return []