autoqild.utilities.statistical_tests

Implementation of paired t-test and wilcoxon_signed_rank_test used to detect leakage using blind classifiers.

Functions

paired_ttest(x1, x2, n_training_folds, ...)

Performs a paired t-test on two sets of values with and without correction.

wilcoxon_signed_rank_test(accuracies, ...[, ...])

Performs the Wilcoxon signed-rank test on two sets of accuracies.
autoqild.utilities.statistical_tests.paired_ttest(x1, x2, n_training_folds, n_test_folds, correction=True, alternative='two-sided', verbose=False)[source]

Performs a paired t-test on two sets of values with and without correction.

Parameters:

  • x1 (ndarray) – First set of values.

  • x2 (ndarray) – Second set of values.

  • n_training_folds (int) – Number of training folds.

  • n_test_folds (int) – Number of test folds.

  • correction (bool, optional) – If True, applies a correction to the variance (default is True).

  • alternative (str, optional) – Defines the alternative hypothesis (default is “two-sided”).

  • verbose (bool, optional) – If True, outputs additional logging information (default is False).

Returns:

p_value – The p-value from the paired t-test.

Return type:

float

autoqild.utilities.statistical_tests.wilcoxon_signed_rank_test(accuracies, accuracies2, alternative='two-sided', verbose=False)[source]

Performs the Wilcoxon signed-rank test on two sets of accuracies.

Parameters:

  • accuracies (ndarray) – First set of accuracy values.

  • accuracies2 (ndarray) – Second set of accuracy values.

  • alternative (str, optional) – Defines the alternative hypothesis (default is “two-sided”).

  • verbose (bool, optional) – If True, outputs additional logging information (default is False).

Returns:

p_value – The p-value from the Wilcoxon signed-rank test.

Return type:

float