tsdat.config.pipeline

Classes

PipelineConfig

Contains configuration parameters for tsdat pipelines.
class tsdat.config.pipeline.PipelineConfig[source]

Bases: tsdat.config.utils.ParameterizedConfigClass, tsdat.config.utils.YamlModel

Contains configuration parameters for tsdat pipelines.

This class is ultimately converted into a tsdat.pipeline.base.Pipeline subclass that will be used to process data.

Provides methods to support yaml parsing and validation, including the generation of json schema for immediate validation. This class also provides a method to instantiate a tsdat.pipeline.base.Pipeline subclass from a parsed configuration file.

Parameters

  • classname (str) – The dotted module path to the pipeline that the specified configurations should apply to. To use the built-in IngestPipeline, for example, you would set ‘tsdat.pipeline.pipelines.IngestPipeline’ as the classname.

  • triggers (List[Pattern[str]]) – A list of regex patterns that should trigger this pipeline when matched with an input key.

  • retriever (Union[Overrideable[RetrieverConfig], RetrieverConfig]) – Either the path to the retriever configuration yaml file and any overrides that should be applied, or the retriever configurations themselves.

  • dataset (Union[Overrideable[DatasetConfig], DatasetConfig]) – Either the path to the dataset configuration yaml file and any overrides that should be applied, or the dataset configurations themselves.

  • quality (Union[Overrideable[QualityConfig], QualityConfig]) – Either the path to the quality configuration yaml file and any overrides that should be applied, or the quality configurations themselves.

  • storage (Union[Overrideable[StorageConfig], StorageConfig]) – Either the path to the storage configuration yaml file and any overrides that should be applied, or the storage configurations themselves.

dataset :Union[tsdat.config.utils.Overrideable[tsdat.config.dataset.DatasetConfig], tsdat.config.dataset.DatasetConfig][source]
quality :Union[tsdat.config.utils.Overrideable[tsdat.config.quality.QualityConfig], tsdat.config.quality.QualityConfig][source]
retriever :Union[tsdat.config.utils.Overrideable[tsdat.config.retriever.RetrieverConfig], tsdat.config.retriever.RetrieverConfig][source]
storage :Union[tsdat.config.utils.Overrideable[tsdat.config.storage.StorageConfig], tsdat.config.storage.StorageConfig][source]
triggers :List[Pattern][source]

Class Methods

instantiate_pipeline

Loads the tsdat.pipeline.BasePipeline subclass specified by the classname property.

merge_overrideable_yaml

Method Descriptions

instantiate_pipeline(self) → tsdat.pipeline.base.Pipeline[source]

Loads the tsdat.pipeline.BasePipeline subclass specified by the classname property.

Properties and sub-properties of the PipelineConfig class that are subclasses of tsdat.config.utils.ParameterizedConfigClass (e.g, classes that define a ‘classname’ and optional ‘parameters’ properties) will also be instantiated in similar fashion. See tsdat.config.utils.recursive_instantiate for implementation details.

Returns

Pipeline – An instance of a tsdat.pipeline.base.Pipeline subclass.

classmethod merge_overrideable_yaml(cls, v: Dict[str, Any], values: Dict[str, Any], field: pydantic.fields.ModelField)[source]