cca_zoo.model_selection.permutation_test_score#

class cca_zoo.model_selection.permutation_test_score(estimator, views, y=None, groups=None, cv=None, n_permutations=100, n_jobs=None, random_state=0, verbose=0, scoring=None, fit_params=None)[source]#

Bases:

Evaluate the significance of a cross-validated score with permutations.

Permutes targets to generate ‘randomized data’ and compute the empirical p-value against the null hypothesis that features and targets are independent. A small p-value suggests that there is a real dependency between features and targets which has been used by the estimator to give good predictions. A large p-value may be due to lack of real dependency between features and targets or the estimator was not able to use the dependency to give good predictions.

Read more in the User Guide.

Parameters:

  • estimator (object) – Estimator object implementing ‘fit’. The object to use to fit the data.

  • views (list or tuple of array-like) – List or tuple of numpy arrays or array-likes with the same number of rows (samples).

  • y (array-like of shape (n_samples,) or (n_samples, n_outputs) or None, optional) – The target variable to try to predict in the case of supervised learning.

  • groups (array-like of shape (n_samples,), optional, default=None) – Labels to constrain permutation within groups. When not specified, y values are permuted among all samples. When a grouped cross-validator is used, the group labels are also passed on to the split method of the cross-validator.

  • scoring (str or callable, optional, default=None) – A single string (see The scoring parameter: defining model evaluation rules) or a callable (see Defining your scoring strategy from metric functions) to evaluate the predictions on the test set. If None the estimator’s score method is used.

  • cv (int, cross-validation generator or an iterable, optional, default=None) – Determines the cross-validation splitting strategy. See notes below for more detail.

  • n_permutations (int, default=100) – Number of times to permute y.

  • n_jobs (int, optional, default=None) – Number of jobs to run in parallel.

  • random_state (int, RandomState instance or None, default=0) – Pass an int for reproducible output for permutation of y values among samples.

  • verbose (int, default=0) – The verbosity level.

  • fit_params (dict, optional, default=None) – Parameters to pass to the fit method of the estimator.

Notes

For cv:

Possible inputs for cv are: - None, to use the default 5-fold cross validation, - int, to specify the number of folds in a (Stratified)KFold, - CV splitter, - An iterable yielding (train, test) splits as arrays of indices. For int/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used. These splitters are instantiated with shuffle=False so the splits will be the same across calls. Refer User Guide for the various cross-validation strategies that can be used here.