tsdat.qc.handlers
¶
Classes¶
Throw an exception, halting the pipeline & indicating a critical error |
|
Symbolic constants used for referencing QC-related |
|
Class containing code to be executed if a particular quality check fails. |
|
Record the results of the quality check in an ancillary qc variable. |
|
Replace all the failed values with _FillValue |
|
Send an email to the recipients using AWS services. |
|
Sort coordinate data using xr.Dataset.sortby(). Accepts the following |
-
class
tsdat.qc.handlers.
FailPipeline
(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityHandler
Throw an exception, halting the pipeline & indicating a critical error
Class Methods
Perform a follow-on action if a quality check fails. This can be used
Method Descriptions
-
run
(self, variable_name: str, results_array: numpy.ndarray)[source]¶ Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).
- Parameters
variable_name (str) – Name of the variable that failed
results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.
-
-
class
tsdat.qc.handlers.
QCParamKeys
[source]¶ Symbolic constants used for referencing QC-related fields in the pipeline config file
-
class
tsdat.qc.handlers.
QualityHandler
(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
abc.ABC
Class containing code to be executed if a particular quality check fails.
- Parameters
ds (xr.Dataset) – The dataset the handler will be applied to
previous_data (xr.Dataset) – A dataset from the previous processing interval (i.e., file). This is used to check for consistency between files, such as for monotonic or delta checks when we need to check the previous value.
quality_manager (QualityManagerDefinition) – The quality_manager definition as specified in the pipeline config file
parameters (dict, optional) – A dictionary of handler-specific parameters specified in the pipeline config file. Defaults to {}
Class Methods
If a correction was made to variable data to fix invalid values
Perform a follow-on action if a quality check fails. This can be used
Method Descriptions
-
record_correction
(self, variable_name: str)[source]¶ If a correction was made to variable data to fix invalid values as detected by a quality check, this method will record the fix to the appropriate variable attribute. The correction description will come from the handler params which get set in the pipeline config file.
- Parameters
variable_name (str) – Name
-
abstract
run
(self, variable_name: str, results_array: numpy.ndarray)[source]¶ Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).
- Parameters
variable_name (str) – Name of the variable that failed
results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.
-
class
tsdat.qc.handlers.
RecordQualityResults
(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityHandler
Record the results of the quality check in an ancillary qc variable.
Class Methods
Perform a follow-on action if a quality check fails. This can be used
Method Descriptions
-
run
(self, variable_name: str, results_array: numpy.ndarray)[source]¶ Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).
- Parameters
variable_name (str) – Name of the variable that failed
results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.
-
-
class
tsdat.qc.handlers.
RemoveFailedValues
(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityHandler
Replace all the failed values with _FillValue
Class Methods
Perform a follow-on action if a quality check fails. This can be used
Method Descriptions
-
run
(self, variable_name: str, results_array: numpy.ndarray)[source]¶ Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).
- Parameters
variable_name (str) – Name of the variable that failed
results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.
-
-
class
tsdat.qc.handlers.
SendEmailAWS
(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityHandler
Send an email to the recipients using AWS services.
Class Methods
Perform a follow-on action if a quality check fails. This can be used
Method Descriptions
-
run
(self, variable_name: str, results_array: numpy.ndarray)[source]¶ Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).
- Parameters
variable_name (str) – Name of the variable that failed
results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.
-
-
class
tsdat.qc.handlers.
SortDatasetByCoordinate
(ds: xarray.Dataset, previous_data: xarray.Dataset, quality_manager: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityHandler
Sort coordinate data using xr.Dataset.sortby(). Accepts the following parameters:
parameters: # Whether or not to sort in ascending order. Defaults to True. ascending: True
Class Methods
Perform a follow-on action if a quality check fails. This can be used
Method Descriptions
-
run
(self, variable_name: str, results_array: numpy.ndarray)[source]¶ Perform a follow-on action if a quality check fails. This can be used to correct data if needed (such as replacing a bad value with missing value, emailing a contact persion, or raising an exception if the failure constitutes a critical error).
- Parameters
variable_name (str) – Name of the variable that failed
results_array (np.ndarray) – An array of True/False values for each data value of the variable. True means the check failed.
-