tsdat.config.pipeline

Classes

PipelineConfig

Class used to contain configuration parameters for tsdat pipelines. This class will

class tsdat.config.pipeline.PipelineConfig

Bases: tsdat.config.utils.ParameterizedConfigClass, tsdat.config.utils.YamlModel

Class used to contain configuration parameters for tsdat pipelines. This class will ultimately be converted into a tsdat.pipeline.base.Pipeline subclass for use in tsdat pipelines.

Provides methods to support yaml parsing and validation, including the generation of json schema for immediate validation. This class also provides a method to instantiate a tsdat.pipeline.base.Pipeline subclass from a parsed configuration file.

Parameters

• classname (str) – The dotted module path to the pipeline that the specified

• should apply to. To use the built-in IngestPipeline (configurations) –

• example (for) –

:param : :param you would set ‘tsdat.pipeline.pipelines.IngestPipeline’ as the classname.: :param triggers: A list of regex patterns that should trigger this :type triggers: List[Pattern[str]] :param pipeline when matched with an input key.: :param retriever: Either the :type retriever: Union[Overrideable[RetrieverConfig], RetrieverConfig] :param path to the retriever configuration yaml file and any overrides that should be: :param applied: :param or the retriever configurations themselves.: :param dataset: Either the path to :type dataset: Union[Overrideable[DatasetConfig], DatasetConfig] :param the dataset configuration yaml file and any overrides that should be applied: :param or: :param the dataset configurations themselves.: :param quality: Either the path to :type quality: Union[Overrideable[QualityConfig], QualityConfig] :param the quality configuration yaml file and any overrides that should be applied: :param or: :param the dataset configurations themselves.: :param storage: Either the path to :type storage: Union[Overrideable[StorageConfig], StorageConfig] :param the storage configuration yaml file and any overrides that should be applied: :param or: :param the storage configurations themselves.:

dataset :Union[tsdat.config.utils.Overrideable[tsdat.config.dataset.DatasetConfig], tsdat.config.dataset.DatasetConfig]

quality :Union[tsdat.config.utils.Overrideable[tsdat.config.quality.QualityConfig], tsdat.config.quality.QualityConfig]

retriever :Union[tsdat.config.utils.Overrideable[tsdat.config.retriever.RetrieverConfig], tsdat.config.retriever.RetrieverConfig]

storage :Union[tsdat.config.utils.Overrideable[tsdat.config.storage.StorageConfig], tsdat.config.storage.StorageConfig]

triggers :List[Pattern]

Class Methods

instantiate_pipeline	This method instantiates the tsdat.pipeline.BasePipeline subclass referenced by the
merge_overrideable_yaml

Method Descriptions

instantiate_pipeline(self) → tsdat.pipeline.base.Pipeline

This method instantiates the tsdat.pipeline.BasePipeline subclass referenced by the classname property on the PipelineConfig instance and passes all properties on the PipelineConfig class (except for ‘classname’) as keyword arguments to the constructor of the tsdat.pipeline.BasePipeline subclass.

Properties and sub-properties of the PipelineConfig class that are subclasses of tsdat.config.utils.ParameterizedConfigClass (e.g, classes that define a ‘classname’ and optional ‘parameters’ properties) will also be instantiated in similar fashion. See tsdat.config.utils.recursive_instantiate for implementation details.

Returns

An instance of a tsdat.pipeline.base.Pipeline subclass.

Return type

Pipeline

classmethod merge_overrideable_yaml(cls, v: Dict[str, Any], values: Dict[str, Any], field: pydantic.fields.ModelField)