tsdat.qc.handlers

Classes

FailPipeline	Throw an exception, halting the pipeline & indicating a critical error
QCParamKeys	Symbolic constants used for referencing QC-related
QualityHandler	Class containing code to be executed if a particular quality check fails.
RecordQualityResults	Record the results of the quality check in an ancillary qc variable.
RemoveFailedValues	Replace all the failed values with _FillValue
SendEmailAWS	Send an email to the recipients using AWS services.
SortDatasetByCoordinate	Sort coordinate data using xr.Dataset.sortby(). Accepts the following

class tsdat.qc.handlers.FailPipeline(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Throw an exception, halting the pipeline & indicating a critical error

Class Methods

run	Perform a follow-on action if a quality check fails. This can be used

Method Descriptions

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.QCParamKeys

Symbolic constants used for referencing QC-related fields in the pipeline config file

ASSESSMENT = assessment

CORRECTION = correction

QC_BIT = bit

TEST_MEANING = meaning

class tsdat.qc.handlers.QualityHandler(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: abc.ABC

Class containing code to be executed if a particular quality check fails.

Parameters

• ds (xr.Dataset) – The dataset the handler will be applied to

• previous_data (xr.Dataset) – A dataset from the previous processing interval (i.e., file). This is used to check for consistency between files, such as for monotonic or delta checks when we need to check the previous value.

• quality_manager (QualityManagerDefinition) – The quality_manager definition as specified in the pipeline config file

• parameters (dict, optional) – A dictionary of handler-specific parameters specified in the pipeline config file. Defaults to {}

Class Methods

record_correction	If a correction was made to variable data to fix invalid values
run	Perform a follow-on action if a quality check fails. This can be used

Method Descriptions

record_correction(self, variable_name: str)

If a correction was made to variable data to fix invalid values as detected by a quality check, this method will record the fix to the appropriate variable attribute. The correction description will come from the handler params which get set in the pipeline config file.

Parameters

variable_name (str) – Name

abstract run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.RecordQualityResults(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Record the results of the quality check in an ancillary qc variable.

Class Methods

run	Perform a follow-on action if a quality check fails. This can be used

Method Descriptions

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.RemoveFailedValues(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Replace all the failed values with _FillValue

Class Methods

run	Perform a follow-on action if a quality check fails. This can be used

Method Descriptions

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.SendEmailAWS(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Send an email to the recipients using AWS services.

Class Methods

run	Perform a follow-on action if a quality check fails. This can be used

Method Descriptions

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.SortDatasetByCoordinate(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Sort coordinate data using xr.Dataset.sortby(). Accepts the following parameters:

parameters:
  # Whether or not to sort in ascending order. Defaults to True.
  ascending: True

Class Methods

run	Perform a follow-on action if a quality check fails. This can be used

Method Descriptions

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.