tsdat.qc.handlers

Module Contents

Classes

QCParamKeys	Symbolic constants used for referencing QC-related
QualityHandler	Class containing code to be executed if a particular quality check fails.
RecordQualityResults	Record the results of the quality check in an ancillary qc variable.
RemoveFailedValues	Replace all the failed values with _FillValue
SortDatasetByCoordinate	Sort coordinate data using xr.Dataset.sortby(). Accepts the following
SendEmailAWS	Send an email to the recipients using AWS services.
FailPipeline	Throw an exception, halting the pipeline & indicating a critical error

class tsdat.qc.handlers.QCParamKeys

Symbolic constants used for referencing QC-related fields in the pipeline config file

QC_BIT = bit

ASSESSMENT = assessment

TEST_MEANING = meaning

CORRECTION = correction

class tsdat.qc.handlers.QualityHandler(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: abc.ABC

Class containing code to be executed if a particular quality check fails.

Parameters

• ds (xr.Dataset) – The dataset the handler will be applied to

• previous_data (xr.Dataset) – A dataset from the previous processing interval (i.e., file). This is used to check for consistency between files, such as for monotonic or delta checks when we need to check the previous value.

• quality_manager (QualityManagerDefinition) – The quality_manager definition as specified in the pipeline config file

• parameters (dict, optional) – A dictionary of handler-specific parameters specified in the pipeline config file. Defaults to {}

abstract run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

record_correction(self, variable_name: str)

If a correction was made to variable data to fix invalid values as detected by a quality check, this method will record the fix to the appropriate variable attribute. The correction description will come from the handler params which get set in the pipeline config file.

Parameters

variable_name (str) – Name

class tsdat.qc.handlers.RecordQualityResults(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Record the results of the quality check in an ancillary qc variable.

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.RemoveFailedValues(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Replace all the failed values with _FillValue

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.SortDatasetByCoordinate(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Sort coordinate data using xr.Dataset.sortby(). Accepts the following parameters:

parameters:
  # Whether or not to sort in ascending order. Defaults to True.
  ascending: True

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.SendEmailAWS(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Send an email to the recipients using AWS services.

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.

class tsdat.qc.handlers.FailPipeline(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)

Bases: QualityHandler

Throw an exception, halting the pipeline & indicating a critical error

run(self, variable_name: str, results_array: numpy.ndarray)

Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).

Parameters

• variable_name (str) – Name of the variable that failed

• results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.