Metrics

dlordinal.metrics.accuracy_off1(y_true: ndarray, y_pred: ndarray, labels=None) → float

Computes the accuracy of the predictions, allowing errors if they occur in an adjacent class.

Parameters:

• y_true (array-like) – Target labels.

• y_pred (array-like) – Predicted probabilities or labels.

• labels (array-like or None) – Labels of the classes. If None, the labels are inferred from the data.

Returns:

acc – 1-off accuracy.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import accuracy_off1
>>> y_true = np.array([0, 0, 1, 2, 3, 0, 0])
>>> y_pred = np.array([0, 1, 1, 2, 0, 0, 1])
>>> accuracy_off1(y_true, y_pred)
0.8571428571428571

dlordinal.metrics.amae(y_true: ndarray, y_pred: ndarray)

Computes the average mean absolute error computed independently for each class as presented in Baccianella et al.[1].

Parameters:

• y_true (array-like) – Targets labels with one-hot or integer encoding.

• y_pred (array-like) – Predicted probabilities or labels.

Returns:

amae – Average mean absolute error.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import amae
>>> y_true = np.array([0, 0, 1, 2, 3, 0, 0])
>>> y_pred = np.array([0, 1, 1, 2, 3, 0, 1])
>>> amae(y_true, y_pred)
0.125

dlordinal.metrics.gmes(y_true: ndarray, y_pred: ndarray) → float

Computes the Geometric Mean of the Sensitivities of the first and last class (GMES). This metric is useful for assessing the balanced performance between the extreme classes using geometric mean.

Parameters:

• y_true (array-like) – Target labels.

• y_pred (array-like) – Predicted probabilities or labels.

Returns:

gmes – Geometric mean of the sensitivities of the extreme classes.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import gmes
>>> y_true = np.array([0, 0, 1, 2, 3, 0, 0])
>>> y_pred = np.array([0, 1, 1, 2, 3, 0, 1])
>>> gmes(y_true, y_pred)
0.7071067811865476

dlordinal.metrics.gmsec(y_true: ndarray, y_pred: ndarray) → float

Geometric Mean of the Sensitivity of the Extreme Classes (GMSEC). It was proposed in (Vargas et al.[2]) with the aim of assessing the performance of the classification performance for the first and the last classes.

Parameters:

• y_true (array-like) – Target labels.

• y_pred (array-like) – Predicted probabilities or labels.

Returns:

gmec – Geometric mean of the sensitivities of the extreme classes.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import gmsec
>>> y_true = np.array([0, 0, 1, 2, 3, 0, 0])
>>> y_pred = np.array([0, 1, 1, 2, 3, 0, 1])
>>> gmsec(y_true, y_pred)
0.7071067811865476

dlordinal.metrics.mes(y_true: ndarray, y_pred: ndarray) → float

Computes the Mean of the Sensitivities of the first and last class (MES). This metric is useful for assessing the balanced performance between the extreme classes using arithmetic mean.

Parameters:

• y_true (array-like) – Target labels.

• y_pred (array-like) – Predicted probabilities or labels.

Returns:

mes – Mean of the sensitivities of the extreme classes.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import mes
>>> y_true = np.array([0, 0, 1, 2, 3, 0, 0])
>>> y_pred = np.array([0, 1, 1, 2, 3, 0, 1])
>>> mes(y_true, y_pred)
0.75

dlordinal.metrics.minimum_sensitivity(y_true: ndarray, y_pred: ndarray) → float

Computes the sensitivity by class and returns the lowest value.

Parameters:

• y_true (array-like) – Target labels.

• y_pred (array-like) – Predicted probabilities or labels.

Returns:

ms – Minimum sensitivity.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import minimum_sensitivity
>>> y_true = np.array([0, 0, 1, 2, 3, 0, 0])
>>> y_pred = np.array([0, 1, 1, 2, 3, 0, 1])
>>> minimum_sensitivity(y_true, y_pred)
0.5

dlordinal.metrics.mmae(y_true: ndarray, y_pred: ndarray)

Computes the maximum mean absolute error computed independently for each class as presented in Cruz-Ramirez et al.[3].

Parameters:

• y_true (array-like) – Target labels with one-hot or integer encoding.

• y_pred (array-like) – Predicted probabilities or labels.

Returns:

mmae – Maximum mean absolute error.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import mmae
>>> y_true = np.array([0, 0, 1, 2, 3, 0, 0])
>>> y_pred = np.array([0, 1, 1, 2, 3, 0, 1])
>>> mmae(y_true, y_pred)
0.5

dlordinal.metrics.ranked_probability_score(y_true, y_proba)

Computes the ranked probability score as presented in Janitza et al.[4].

Parameters:

• y_true (array-like) – Target labels.

• y_proba (array-like) – Predicted probabilities.

Returns:

rps – The ranked probability score.

Return type:

float

Examples

>>> import numpy as np
>>> from dlordinal.metrics import ranked_probability_score
>>> y_true = np.array([0, 0, 3, 2])
>>> y_pred = np.array([[0.2, 0.4, 0.2, 0.2], [0.7, 0.1, 0.1, 0.1], [0.5, 0.05, 0.1, 0.35], [0.1, 0.05, 0.65, 0.2]])
>>> ranked_probability_score(y_true, y_pred)
0.5068750000000001

dlordinal.metrics.write_array_to_file(array: ndarray, path_str: str, id: str)

Writes an array to a json file. The array is saved as a dictionary with the key ‘id’ and the value ‘array’.

Parameters:

• array (array-like) – Array to be saved.

• path_str (str) – Path to the file that will be saved. The directory of the file will be created if it does not exist.

• id (str) – Id of the array.

Examples

>>> array = np.array([0, 1, 2])
>>> write_array_to_file(array, 'results.json', 'array')
>>> with open('results.json', 'r') as f:
...     print(f.read())
{"array": [0, 1, 2]}

>>> array2 = np.array([3, 4, 5])
>>> write_array_to_file(array, 'results.json', 'array2')
>>> with open('results.json', 'r') as f:
...     print(f.read())
{"array": [0, 1, 2], "array2": [3, 4, 5]}

dlordinal.metrics.write_metrics_dict_to_file(metrics: ~typing.Dict[str, float], path_str: str, filter_fn: ~typing.Callable[[str, float], bool] | None = <function <lambda>>) → None

Writes a dictionary of metrics to a tabular file. The dictionary is filtered by the filter function. The first time that the metrics are saved to the file, the keys are written as the header. Subsequent calls append the values to the file.

Parameters:

• metrics (Dict[str, float]) – Dictionary of metric names associated with their value.

• path_str (str) – Path to the file that will be saved. The directory of the file will be created if it does not exist. If the file exists, the metrics will be appended to the file in a new row.

• filter_fn (Optional[Callable[[str, bool], bool]], default=lambda n, v: True) – Function that filters the metrics. The function takes the name and the value of the metric and returns True if the metric should be saved.

Examples

>>> metrics = {'acc': 0.5, 'gmsec': 0.25}
>>> write_metrics_dict_to_file(metrics, 'results.txt')
>>> write_metrics_dict_to_file(metrics, 'results.txt')
>>> with open('results.txt', 'r') as f:
...     print(f.read())
acc gmsec
0.5 0.25
0.5 0.25

>>> write_metrics_dict_to_file(metrics, 'results.txt', filter_fn=lambda name, value: name == 'acc')
>>> with open('results.txt', 'r') as f:
...     print(f.read())
acc
0.5
0.5

[1]

Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Evaluation measures for ordinal regression. In 2009 Ninth international conference on intelligent systems design and applications, 283–287. IEEE, 2009. doi:10.1109/ISDA.2009.230.

[2]

Víctor Manuel Vargas, Pedro Antonio Gutiérrez, Javier Barbero-Gómez, and César Hervás-Martínez. Improving the classification of extreme classes by means of loss regularisation and generalised beta distributions. 2024. URL: https://arxiv.org/abs/2407.12417, arXiv:2407.12417.

[3]

Manuel Cruz-Ramirez, Cesar Hervas-Martinez, Javier Sanchez-Monedero, and Pedro Antonio Gutierrez. Metrics to guide a multi-objective evolutionary algorithm for ordinal classification. Neurocomputing, 135:21–31, 2014. doi:10.1016/j.neucom.2013.05.058.

[4]

Silke Janitza, Gerhard Tutz, and Anne-Laure Boulesteix. Random forest for ordinal responses: prediction and variable selection. Computational Statistics & Data Analysis, 96:57–73, 2016. doi:https://doi.org/10.1016/j.csda.2015.10.005.