tsdat.io.readers

Classes

CSVReader	Uses pandas and xarray functions to read a csv file and extract its contents into an
NetCDFReader	Thin wrapper around xarray's open_dataset() function, with optional parameters
ParquetReader	Uses pandas and xarray functions to read a parquet file and extract its contents
ZarrReader	Uses xarray's Zarr capabilities to read a Zarr archive and extract its contents into
ZipReader	DataReader for reading from a zipped archive. Writing to this format is not supported.

class tsdat.io.readers.CSVReader

Bases: tsdat.io.base.DataReader

Uses pandas and xarray functions to read a csv file and extract its contents into an xarray Dataset object. Two parameters are supported: read_csv_kwargs and from_dataframe_kwargs, whose contents are passed as keyword arguments to pandas.read_csv() and xarray.Dataset.from_dataframe() respectively.

class Parameters

Bases: pydantic.BaseModel

from_dataframe_kwargs :Dict[str, Any]

read_csv_kwargs :Dict[str, Any]

parameters :CSVReader.Parameters

Class Methods

read	Reads data given an input key.

Method Descriptions

read(self, input_key: str) → xarray.Dataset

Reads data given an input key.

Uses the input key to open a resource and load data as a xr.Dataset object or as a mapping of strings to xr.Dataset objects.

In most cases DataReaders will only need to return a single xr.Dataset, but occasionally some types of inputs necessitate that the data loaded from the input_key be returned as a mapping. For example, if the input_key is a path to a zip file containing multiple disparate datasets, then returning a mapping is appropriate.

Parameters

input_key (str) – An input key matching the DataReader’s regex pattern that should be used to load data.

Returns

Union[xr.Dataset, Dict[str, xr.Dataset]] –

The raw data extracted from the

provided input key.

class tsdat.io.readers.NetCDFReader

Bases: tsdat.io.base.DataReader

Thin wrapper around xarray’s open_dataset() function, with optional parameters used as keyword arguments in the function call.

parameters :Dict[str, Any]

Class Methods

read	Reads data given an input key.

Method Descriptions

read(self, input_key: str) → xarray.Dataset

Reads data given an input key.

Uses the input key to open a resource and load data as a xr.Dataset object or as a mapping of strings to xr.Dataset objects.

In most cases DataReaders will only need to return a single xr.Dataset, but occasionally some types of inputs necessitate that the data loaded from the input_key be returned as a mapping. For example, if the input_key is a path to a zip file containing multiple disparate datasets, then returning a mapping is appropriate.

Parameters

input_key (str) – An input key matching the DataReader’s regex pattern that should be used to load data.

Returns

Union[xr.Dataset, Dict[str, xr.Dataset]] –

The raw data extracted from the

provided input key.

class tsdat.io.readers.ParquetReader

Bases: tsdat.io.base.DataReader

Uses pandas and xarray functions to read a parquet file and extract its contents into an xarray Dataset object. Two parameters are supported: read_parquet_kwargs and from_dataframe_kwargs, whose contents are passed as keyword arguments to pandas.read_parquet() and xarray.Dataset.from_dataframe() respectively.

class Parameters

Bases: pydantic.BaseModel

from_dataframe_kwargs :Dict[str, Any]

read_parquet_kwargs :Dict[str, Any]

parameters :ParquetReader.Parameters

Class Methods

read	Reads data given an input key.

Method Descriptions

read(self, input_key: str) → xarray.Dataset

Reads data given an input key.

Uses the input key to open a resource and load data as a xr.Dataset object or as a mapping of strings to xr.Dataset objects.

In most cases DataReaders will only need to return a single xr.Dataset, but occasionally some types of inputs necessitate that the data loaded from the input_key be returned as a mapping. For example, if the input_key is a path to a zip file containing multiple disparate datasets, then returning a mapping is appropriate.

Parameters

input_key (str) – An input key matching the DataReader’s regex pattern that should be used to load data.

Returns

Union[xr.Dataset, Dict[str, xr.Dataset]] –

The raw data extracted from the

provided input key.

class tsdat.io.readers.ZarrReader

Bases: tsdat.io.base.DataReader

Uses xarray’s Zarr capabilities to read a Zarr archive and extract its contents into an xarray Dataset object.

class Parameters

Bases: pydantic.BaseModel

open_zarr_kwargs :Dict[str, Any]

parameters :ZarrReader.Parameters

Class Methods

read	Reads data given an input key.

Method Descriptions

read(self, input_key: str) → xarray.Dataset

Reads data given an input key.

Uses the input key to open a resource and load data as a xr.Dataset object or as a mapping of strings to xr.Dataset objects.

In most cases DataReaders will only need to return a single xr.Dataset, but occasionally some types of inputs necessitate that the data loaded from the input_key be returned as a mapping. For example, if the input_key is a path to a zip file containing multiple disparate datasets, then returning a mapping is appropriate.

Parameters

input_key (str) – An input key matching the DataReader’s regex pattern that should be used to load data.

Returns

Union[xr.Dataset, Dict[str, xr.Dataset]] –

The raw data extracted from the

provided input key.

class tsdat.io.readers.ZipReader(parameters: Dict = None)

Bases: tsdat.io.base.ArchiveReader

DataReader for reading from a zipped archive. Writing to this format is not supported.

This class requires a that `readers be specified in the parameters section of the storage configuration file. The structure of the `readers section should mirror the structure of its parent `readers section. To illustrate, consider the following configuration block:

readers:
  .*:
    zip:
      file_pattern: '.*\.zip'
      classname: "tsdat.io.readers.ZipReader"
      parameters:
        # Parameters to specify how the ZipReader should read/unpack the archive.
        # Parameters here are passed to the Python open() method as kwargs. The
        # default value is shown below.
        open_zip_kwargs:
          mode: "rb"

        # Parameters here are passed to zipfile.ZipFile.open() as kwargs. Useful
        # for specifying the system encoding or compression algorithm to use for
        # unpacking the archive. These are optional.
        read_zip_kwargs:
          mode: "r"


        # The readers section tells the ZipReaders which DataReaders should be
        # used to read the unpacked files.
        readers:
          r".*\.csv":
            classname: tsdat.io.readers.CSVReader
            parameters:  # Parameters specific to tsdat.io.readers.CsvReader
                read_csv_kwargs:
                sep: '\t'

        # Pattern(s) used to exclude certain files in the archive from being handled.
        # This parameter is optional, and the default value is shown below:
        exclude: ['.*\_\_MACOSX/.*', '.*\.DS_Store']

class Parameters

Bases: pydantic.BaseModel

exclude :List[str] = []

open_zip_kwargs :Dict[str, Any]

read_zip_kwargs :Dict[str, Any]

readers :Dict[str, Any]

parameters :ZipReader.Parameters

Class Methods

read	Extracts the file into memory and uses registered DataReaders to read each relevant

Method Descriptions

read(self, input_key: str) → Dict[str, xarray.Dataset]

Extracts the file into memory and uses registered DataReaders to read each relevant extracted file into its own xarray Dataset object. Returns a mapping like {filename: xr.Dataset}.

Parameters

• input_key (Union[str, BytesIO]) – The file to read in. Can be provided as a filepath or

• file. (a bytes-like object. It is used to open the zip) –

• name (str, optional) – A label used to help trace the origin of the data read-in.

• file (It is used in the key in the returned dictionary. Must be provided if the) –

• then (argument is not string-like. If file is a string and name is not specified) –

• None. (the label will be set by file. Defaults to) –

Returns

Dict[str, xr.Dataset] – A mapping of {label: xr.Dataset}.