tsdat.config.pipeline
Classes
Contains configuration parameters for tsdat pipelines. |
- class tsdat.config.pipeline.PipelineConfig[source]
Bases:
tsdat.config.utils.ParameterizedConfigClass
,tsdat.config.utils.YamlModel
Contains configuration parameters for tsdat pipelines.
This class is ultimately converted into a tsdat.pipeline.base.Pipeline subclass that will be used to process data.
Provides methods to support yaml parsing and validation, including the generation of json schema for immediate validation. This class also provides a method to instantiate a tsdat.pipeline.base.Pipeline subclass from a parsed configuration file.
- Parameters:
classname (str) – The dotted module path to the pipeline that the specified configurations should apply to. To use the built-in IngestPipeline, for example, you would set ‘tsdat.pipeline.pipelines.IngestPipeline’ as the classname.
triggers (List[Pattern[str]]) – A list of regex patterns that should trigger this pipeline when matched with an input key.
retriever (Union[Overrideable[RetrieverConfig], RetrieverConfig]) – Either the path to the retriever configuration yaml file and any overrides that should be applied, or the retriever configurations themselves.
dataset (Union[Overrideable[DatasetConfig], DatasetConfig]) – Either the path to the dataset configuration yaml file and any overrides that should be applied, or the dataset configurations themselves.
quality (Union[Overrideable[QualityConfig], QualityConfig]) – Either the path to the quality configuration yaml file and any overrides that should be applied, or the quality configurations themselves.
storage (Union[Overrideable[StorageConfig], StorageConfig]) – Either the path to the storage configuration yaml file and any overrides that should be applied, or the storage configurations themselves.
- dataset: tsdat.config.utils.Overrideable[tsdat.config.dataset.DatasetConfig] | tsdat.config.dataset.DatasetConfig[source]
- quality: tsdat.config.utils.Overrideable[tsdat.config.quality.QualityConfig] | tsdat.config.quality.QualityConfig[source]
- retriever: tsdat.config.utils.Overrideable[tsdat.config.retriever.RetrieverConfig] | tsdat.config.retriever.RetrieverConfig[source]
- storage: tsdat.config.utils.Overrideable[tsdat.config.storage.StorageConfig] | tsdat.config.storage.StorageConfig[source]
Class Methods
Loads the tsdat.pipeline.BasePipeline subclass specified by the classname property.
Method Descriptions
- instantiate_pipeline() tsdat.pipeline.base.Pipeline [source]
Loads the tsdat.pipeline.BasePipeline subclass specified by the classname property.
Properties and sub-properties of the PipelineConfig class that are subclasses of tsdat.config.utils.ParameterizedConfigClass (e.g, classes that define a ‘classname’ and optional ‘parameters’ properties) will also be instantiated in similar fashion. See tsdat.config.utils.recursive_instantiate for implementation details.
- Returns:
Pipeline – An instance of a tsdat.pipeline.base.Pipeline subclass.