tsdat.pipeline.base
¶
Classes¶
Base class for tsdat data pipelines. |
-
class
tsdat.pipeline.base.
Pipeline
[source]¶ Bases:
tsdat.utils.ParameterizedClass
,abc.ABC
Base class for tsdat data pipelines.
-
dataset_config
:tsdat.config.dataset.DatasetConfig[source]¶ Describes the structure and metadata of the output dataset.
-
quality
:tsdat.qc.base.QualityManagement[source]¶ Manages the dataset quality through checks and corrections.
-
triggers
:List[Pattern] = [][source]¶ Regex patterns matching input keys to determine when the pipeline should run.
Class Methods
Modifies the retrieved dataset by dropping variables not declared in the
Runs the data pipeline on the provided inputs.
Method Descriptions
-
prepare_retrieved_dataset
(self, dataset: xarray.Dataset) → xarray.Dataset[source]¶ Modifies the retrieved dataset by dropping variables not declared in the DatasetConfig, adding static variables, initializing non-retrieved variables, and importing global and variable-level attributes from the DatasetConfig.
- Parameters
dataset (xr.Dataset) – The retrieved dataset.
- Returns
The dataset with structure and metadata matching the DatasetConfig.
- Return type
xr.Dataset
-
abstract
run
(self, inputs: List[str], **kwargs: Any) → Any[source]¶ Runs the data pipeline on the provided inputs.
- Parameters
inputs (List[str]) – A list of input keys that the pipeline’s Retriever class
use to load data into the pipeline. (can) –
- Returns
The processed dataset.
- Return type
xr.Dataset
-