tsdat.qc.checkers
¶
Classes¶
Check that no values for the specified variable greater less than |
|
Check that no values for the specified variable are less than |
|
Check that no values for the specified variable are greater than |
|
Check that no values for the specified variable are less than |
|
Checks if any values are assigned to _FillValue or ‘NaN’ (for non-time |
|
Checks that all values for the specified variable are either |
|
Check that the difference between any two consecutive |
|
Check that no values for the specified variable are greater than |
|
Check that no values for the specified variable are less than |
|
Check that no values for the specified variable are greater than |
|
Check that no values for the specified variable are less than |
|
Class containing the code to perform a single Quality Check on a |
-
class
tsdat.qc.checkers.
CheckFailMax
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters)[source]¶ Bases:
CheckMax
Check that no values for the specified variable greater less than the maximum vaue set by the ‘fail_range’ attribute. If the variable in question does not posess the ‘fail_range’ attribute, this check will be skipped.
-
class
tsdat.qc.checkers.
CheckFailMin
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters)[source]¶ Bases:
CheckMin
Check that no values for the specified variable are less than the minimum vaue set by the ‘fail_range’ attribute. If the variable in question does not posess the ‘fail_range’ attribute, this check will be skipped.
-
class
tsdat.qc.checkers.
CheckMax
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityChecker
Check that no values for the specified variable are greater than a specified maximum threshold. The threshold value is an attribute set on the variable in question. The attribute name is specified in the quality checker definition in the pipeline config file by setting a param called ‘key: ATTRIBUTE_NAME’.
If the key parameter is not set or the variable does not possess the specified attribute, this check will be skipped.
Class Methods
Check a dataset’s variable to see if it passes a quality check.
Method Descriptions
-
run
(self, variable_name: str) → Optional[numpy.ndarray][source]¶ Check a dataset’s variable to see if it passes a quality check. These checks can be performed on the entire variable at one time by using xarray vectorized numerical operators.
- Parameters
variable_name (str) – The name of the variable to check
- Returns
If the check was performed, return a ndarray of the same shape as the variable. Each value in the data array will be either True or False, depending upon the results of the check. True means the check failed. False means it succeeded.
Note that we are using an np.ndarray instead of an xr.DataArray because the DataArray contains coordinate indexes which can sometimes get out of sync when performing np arithmectic vector operations. So it’s easier to just use numpy arrays.
If the check was skipped for some reason (i.e., it was not relevant given the current attributes defined for this dataset), then the run method should return None.
- Return type
Optional[np.ndarray]
-
-
class
tsdat.qc.checkers.
CheckMin
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityChecker
Check that no values for the specified variable are less than a specified minimum threshold. The threshold value is an attribute set on the variable in question. The attribute name is specified in the quality checker definition in the pipeline config file by setting a param called ‘key: ATTRIBUTE_NAME’.
If the key parameter is not set or the variable does not possess the specified attribute, this check will be skipped.
Class Methods
Check a dataset’s variable to see if it passes a quality check.
Method Descriptions
-
run
(self, variable_name: str) → Optional[numpy.ndarray][source]¶ Check a dataset’s variable to see if it passes a quality check. These checks can be performed on the entire variable at one time by using xarray vectorized numerical operators.
- Parameters
variable_name (str) – The name of the variable to check
- Returns
If the check was performed, return a ndarray of the same shape as the variable. Each value in the data array will be either True or False, depending upon the results of the check. True means the check failed. False means it succeeded.
Note that we are using an np.ndarray instead of an xr.DataArray because the DataArray contains coordinate indexes which can sometimes get out of sync when performing np arithmectic vector operations. So it’s easier to just use numpy arrays.
If the check was skipped for some reason (i.e., it was not relevant given the current attributes defined for this dataset), then the run method should return None.
- Return type
Optional[np.ndarray]
-
-
class
tsdat.qc.checkers.
CheckMissing
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityChecker
Checks if any values are assigned to _FillValue or ‘NaN’ (for non-time variables) or checks if values are assigned to ‘NaT’ (for time variables). Also, for non-time variables, checks if values are above or below valid_range, as this is considered missing as well.
Class Methods
Check a dataset’s variable to see if it passes a quality check.
Method Descriptions
-
run
(self, variable_name: str) → Optional[numpy.ndarray][source]¶ Check a dataset’s variable to see if it passes a quality check. These checks can be performed on the entire variable at one time by using xarray vectorized numerical operators.
- Parameters
variable_name (str) – The name of the variable to check
- Returns
If the check was performed, return a ndarray of the same shape as the variable. Each value in the data array will be either True or False, depending upon the results of the check. True means the check failed. False means it succeeded.
Note that we are using an np.ndarray instead of an xr.DataArray because the DataArray contains coordinate indexes which can sometimes get out of sync when performing np arithmectic vector operations. So it’s easier to just use numpy arrays.
If the check was skipped for some reason (i.e., it was not relevant given the current attributes defined for this dataset), then the run method should return None.
- Return type
Optional[np.ndarray]
-
-
class
tsdat.qc.checkers.
CheckMonotonic
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityChecker
Checks that all values for the specified variable are either strictly increasing or strictly decreasing.
Class Methods
Check a dataset’s variable to see if it passes a quality check.
Method Descriptions
-
run
(self, variable_name: str) → Optional[numpy.ndarray][source]¶ Check a dataset’s variable to see if it passes a quality check. These checks can be performed on the entire variable at one time by using xarray vectorized numerical operators.
- Parameters
variable_name (str) – The name of the variable to check
- Returns
If the check was performed, return a ndarray of the same shape as the variable. Each value in the data array will be either True or False, depending upon the results of the check. True means the check failed. False means it succeeded.
Note that we are using an np.ndarray instead of an xr.DataArray because the DataArray contains coordinate indexes which can sometimes get out of sync when performing np arithmectic vector operations. So it’s easier to just use numpy arrays.
If the check was skipped for some reason (i.e., it was not relevant given the current attributes defined for this dataset), then the run method should return None.
- Return type
Optional[np.ndarray]
-
-
class
tsdat.qc.checkers.
CheckValidDelta
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
QualityChecker
Check that the difference between any two consecutive values is not greater than the threshold set by the ‘valid_delta’ attribute. If the variable in question does not posess the ‘valid_delta’ attribute, this check will be skipped.
Class Methods
Check a dataset’s variable to see if it passes a quality check.
Method Descriptions
-
run
(self, variable_name: str) → Optional[numpy.ndarray][source]¶ Check a dataset’s variable to see if it passes a quality check. These checks can be performed on the entire variable at one time by using xarray vectorized numerical operators.
- Parameters
variable_name (str) – The name of the variable to check
- Returns
If the check was performed, return a ndarray of the same shape as the variable. Each value in the data array will be either True or False, depending upon the results of the check. True means the check failed. False means it succeeded.
Note that we are using an np.ndarray instead of an xr.DataArray because the DataArray contains coordinate indexes which can sometimes get out of sync when performing np arithmectic vector operations. So it’s easier to just use numpy arrays.
If the check was skipped for some reason (i.e., it was not relevant given the current attributes defined for this dataset), then the run method should return None.
- Return type
Optional[np.ndarray]
-
-
class
tsdat.qc.checkers.
CheckValidMax
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters)[source]¶ Bases:
CheckMax
Check that no values for the specified variable are greater than the maximum vaue set by the ‘valid_range’ attribute. If the variable in question does not posess the ‘valid_range’ attribute, this check will be skipped.
-
class
tsdat.qc.checkers.
CheckValidMin
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters)[source]¶ Bases:
CheckMin
Check that no values for the specified variable are less than the minimum vaue set by the ‘valid_range’ attribute. If the variable in question does not posess the ‘valid_range’ attribute, this check will be skipped.
-
class
tsdat.qc.checkers.
CheckWarnMax
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters)[source]¶ Bases:
CheckMax
Check that no values for the specified variable are greater than the maximum vaue set by the ‘warn_range’ attribute. If the variable in question does not posess the ‘warn_range’ attribute, this check will be skipped.
-
class
tsdat.qc.checkers.
CheckWarnMin
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters)[source]¶ Bases:
CheckMin
Check that no values for the specified variable are less than the minimum vaue set by the ‘warn_range’ attribute. If the variable in question does not posess the ‘warn_range’ attribute, this check will be skipped.
-
class
tsdat.qc.checkers.
QualityChecker
(ds: xarray.Dataset, previous_data: xarray.Dataset, definition: tsdat.config.QualityManagerDefinition, parameters: Union[Dict, None] = None)[source]¶ Bases:
abc.ABC
Class containing the code to perform a single Quality Check on a Dataset variable.
- Parameters
ds (xr.Dataset) – The dataset the checker will be applied to
previous_data (xr.Dataset) – A dataset from the previous processing interval (i.e., file). This is used to check for consistency between files, such as for monitonic or delta checks when we need to check the previous value.
definition (QualityManagerDefinition) – The quality manager definition as specified in the pipeline config file
parameters (dict, optional) – A dictionary of checker-specific parameters specified in the pipeline config file. Defaults to {}
Class Methods
Check a dataset’s variable to see if it passes a quality check.
Method Descriptions
-
abstract
run
(self, variable_name: str) → Optional[numpy.ndarray][source]¶ Check a dataset’s variable to see if it passes a quality check. These checks can be performed on the entire variable at one time by using xarray vectorized numerical operators.
- Parameters
variable_name (str) – The name of the variable to check
- Returns
If the check was performed, return a ndarray of the same shape as the variable. Each value in the data array will be either True or False, depending upon the results of the check. True means the check failed. False means it succeeded.
Note that we are using an np.ndarray instead of an xr.DataArray because the DataArray contains coordinate indexes which can sometimes get out of sync when performing np arithmectic vector operations. So it’s easier to just use numpy arrays.
If the check was skipped for some reason (i.e., it was not relevant given the current attributes defined for this dataset), then the run method should return None.
- Return type
Optional[np.ndarray]