Deep Models

Models

class cca_zoo.deepmodels.DCCA(latent_dims: int, objective=<class 'cca_zoo.deepmodels.objectives.MCCA'>, encoders=None, r: float = 0, eps: float = 1e-05, **kwargs)

Bases: _BaseDeep, _BaseCCA

A class used to fit a DCCA model.

Citation

Andrew, Galen, et al. “Deep canonical correlation analysis.” International conference on machine learning. PMLR, 2013.

Constructor class for DCCA

Parameters

• latent_dims – # latent dimensions

• objective – # CCA objective: normal tracenorm CCA by default

• encoders – list of encoder networks

• r – regularisation parameter of tracenorm CCA like ridge CCA. Needs to be VERY SMALL. If you get errors make this smaller

• eps – epsilon used throughout. Needs to be VERY SMALL. If you get errors make this smaller

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Define the loss function for the model. This is used by the DeepWrapper class

Parameters

views –

Returns

post_transform(z, train=False)

Some models require a final linear CCA after model training.

Parameters

• z – a list of all of the latent space embeddings for each view

• train – if the train flag is True this fits a new post transformation

pairwise_correlations(loader: DataLoader, train=False)

Calculates correlation for entire batch from dataloader

Parameters

• loader – a dataloader that matches the structure of that used for training

• train – whether to fit final linear transformation

Returns

by default returns the average pairwise correlation in each dimension (for 2 views just the correlation)

score(loader: DataLoader, train=False)

Returns average correlation in each dimension (averages over all pairs for multiview)

Parameters

• loader – a dataloader that matches the structure of that used for training

• train – whether to fit final linear transformation

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

class cca_zoo.deepmodels.DCCAE(latent_dims: int, objective=<class 'cca_zoo.deepmodels.objectives.MCCA'>, encoders=None, decoders=None, r: float = 0, eps: float = 1e-05, lam=0.5, latent_dropout=0, img_dim=None, recon_loss_type='mse', **kwargs)

Bases: DCCA, _GenerativeMixin

A class used to fit a DCCAE model.

Citation

Wang, Weiran, et al. “On deep multi-view representation learning.” International conference on machine learning. PMLR, 2015.

Constructor class for DCCAE

Parameters

• latent_dims – # latent dimensions

• objective – # CCA objective: normal tracenorm CCA by default

• encoders – list of encoder networks

• decoders – list of decoder networks

• r – regularisation parameter of tracenorm CCA like ridge CCA. Needs to be VERY SMALL. If you get errors make this smaller

• eps – epsilon used throughout. Needs to be VERY SMALL. If you get errors make this smaller

• lam – weight of reconstruction loss (1 minus weight of correlation loss)

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Define the loss function for the model. This is used by the DeepWrapper class

Parameters

views –

Returns

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

class cca_zoo.deepmodels.DCCA_NOI(latent_dims: int, N: int, encoders=None, r: float = 0, rho: float = 0.2, eps: float = 1e-09, shared_target: bool = False, **kwargs)

Bases: DCCA

A class used to fit a DCCA model by non-linear orthogonal iterations

Citation

Wang, Weiran, et al. “Stochastic optimization for deep CCA via nonlinear orthogonal iterations.” 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 2015.

Constructor class for DCCA_NOI

Parameters

• latent_dims – # latent dimensions

• N – # samples used to estimate covariance

• encoders – list of encoder networks

• r – regularisation parameter of tracenorm CCA like ridge CCA

• rho – covariance memory like DCCA non-linear orthogonal iterations paper

• eps – epsilon used throughout

• shared_target – not used

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Define the loss function for the model. This is used by the DeepWrapper class

Parameters

views –

Returns

class cca_zoo.deepmodels.DCCA_SDL(latent_dims: int, N: int, encoders=None, r: float = 0, rho: float = 0.2, eps: float = 1e-05, shared_target: bool = False, lam=0.5, **kwargs)

Bases: DCCA_NOI

A class used to fit a Deep CCA by Stochastic Decorrelation model.

Citation

Chang, Xiaobin, Tao Xiang, and Timothy M. Hospedales. “Scalable and effective deep CCA via soft decorrelation.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

Constructor class for DCCA_SDL

Parameters

• latent_dims – # latent dimensions

• encoders – list of encoder networks

• r – regularisation parameter of tracenorm CCA like ridge CCA

• rho – covariance memory like DCCA non-linear orthogonal iterations paper

• eps – epsilon used throughout

• shared_target – not used

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Define the loss function for the model. This is used by the DeepWrapper class

Parameters

views –

Returns

class cca_zoo.deepmodels.DVCCA(latent_dims: int, encoders=None, decoders=None, private_encoders: Optional[Iterable[_BaseEncoder]] = None, latent_dropout=0, img_dim=None, recon_loss_type='mse', **kwargs)

Bases: _BaseDeep, _GenerativeMixin

A class used to fit a DVCCA model.

Citation

Wang, Weiran, et al. ‘Deep variational canonical correlation analysis.’ arXiv preprint arXiv:1610.03454 (2016).

https: // arxiv.org / pdf / 1610.03454.pdf

https: // github.com / pytorch / examples / blob / master / vae / main.py

Constructor class for DVCCA

Parameters

• latent_dims – # latent dimensions

• encoders – list of encoder networks

• decoders – list of decoder networks

• private_encoders – list of private (view specific) encoder networks

forward(views, mle=True, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Loss for model

Parameters

• views –

• args –

• kwargs –

Returns

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

class cca_zoo.deepmodels.BarlowTwins(latent_dims: int, encoders=None, lam=1, **kwargs)

Bases: DCCA

A class used to fit a Barlow Twins model.

Citation

Zbontar, Jure, et al. “Barlow twins: Self-supervised learning via redundancy reduction.” arXiv preprint arXiv:2103.03230 (2021).

Constructor class for Barlow Twins

Parameters

• latent_dims – # latent dimensions

• encoders – list of encoder networks

• lam – weighting of off diagonal loss terms

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Define the loss function for the model. This is used by the DeepWrapper class

Parameters

views –

Returns

class cca_zoo.deepmodels.DTCCA(latent_dims: int, encoders=None, r: float = 0, eps: float = 1e-05, **kwargs)

Bases: DCCA

A class used to fit a DTCCA model.

Is just a thin wrapper round DCCA with the DTCCA objective and a TCCA post-processing

Citation

Wong, Hok Shing, et al. “Deep Tensor CCA for Multi-view Learning.” IEEE Transactions on Big Data (2021).

Constructor class for DTCCA

Parameters

• latent_dims – # latent dimensions

• encoders – list of encoder networks

• r – regularisation parameter of tracenorm CCA like ridge CCA. Needs to be VERY SMALL. If you get errors make this smaller

• eps – epsilon used throughout. Needs to be VERY SMALL. If you get errors make this smaller

post_transform(z, train=False) → Iterable[ndarray]

Some models require a final linear CCA after model training.

Parameters

• z – a list of all of the latent space embeddings for each view

• train – if the train flag is True this fits a new post transformation

class cca_zoo.deepmodels.SplitAE(latent_dims: int, encoder: ~cca_zoo.deepmodels.architectures._BaseEncoder = <class 'cca_zoo.deepmodels.architectures.Encoder'>, decoders=None, latent_dropout=0, recon_loss_type='mse', img_dim=None, **kwargs)

Bases: _BaseDeep, _GenerativeMixin

A class used to fit a Split Autoencoder model.

Citation

Ngiam, Jiquan, et al. “Multimodal deep learning.” ICML. 2011.

Parameters

• latent_dims – # latent dimensions

• encoder – list of encoder networks

• decoders – list of decoder networks

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Loss for model

Parameters

• views –

• args –

• kwargs –

Returns

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

cca_zoo.deepmodels.get_dataloaders(dataset, val_dataset=None, batch_size=None, val_batch_size=None, drop_last=True, val_drop_last=False, shuffle_train=False, pin_memory=True, num_workers=0)

A utility function to allow users to quickly get hold of the dataloaders required by pytorch lightning

Parameters

• dataset – A CCA dataset used for training

• val_dataset – An optional CCA dataset used for validation

• batch_size – batch size of train loader

• val_batch_size – batch size of val loader

• num_workers – number of workers used

• pin_memory – pin memory used by pytorch - True tends to speed up training

• shuffle_train – whether to shuffle training data

• val_drop_last – whether to drop the last incomplete batch from the validation data

• drop_last – whether to drop the last incomplete batch from the train data

Deep Objectives

class cca_zoo.deepmodels.objectives.MCCA(latent_dims: int, r: float = 0, eps: float = 0.001)

Bases: object

Differentiable MCCA Loss. Loss() method takes the outputs of each view’s network and solves the multiset eigenvalue problem as in e.g. https://arxiv.org/pdf/2005.11914.pdf

Parameters

• latent_dims – the number of latent dimensions

• r – regularisation as in regularized CCA. Makes the problem well posed when batch size is similar to

the number of latent dimensions :param eps: an epsilon parameter used in some operations

class cca_zoo.deepmodels.objectives.GCCA(latent_dims: int, r: float = 0, eps: float = 0.001)

Bases: object

Differentiable GCCA Loss. Loss() method takes the outputs of each view’s network and solves the generalized CCA eigenproblem as in https://arxiv.org/pdf/2005.11914.pdf

Parameters

• latent_dims – the number of latent dimensions

• r – regularisation as in regularized CCA. Makes the problem well posed when batch size is similar to

the number of latent dimensions :param eps: an epsilon parameter used in some operations

class cca_zoo.deepmodels.objectives.CCA(latent_dims: int, r: float = 0, eps: float = 0.001)

Bases: object

Differentiable CCA Loss. Loss() method takes the outputs of each view’s network and solves the CCA problem as in Andrew’s original paper

Parameters

• latent_dims – the number of latent dimensions

• r – regularisation as in regularized CCA. Makes the problem well posed when batch size is similar to the number of latent dimensions

• eps – an epsilon parameter used in some operations

class cca_zoo.deepmodels.objectives.TCCA(latent_dims: int, r: float = 0, eps: float = 0.0001)

Bases: object

Differentiable TCCA Loss.

Parameters

• latent_dims – the number of latent dimensions

• r – regularisation as in regularized CCA. Makes the problem well posed when batch size is similar to the number of latent dimensions

• eps – an epsilon parameter used in some operations

Callbacks

class cca_zoo.deepmodels._callbacks.CorrelationCallback

Bases: Callback

on_train_epoch_end(trainer: Trainer, pl_module: LightningModule) → None

Called when the train epoch ends.

To access all batch outputs at the end of the epoch, either:

1. Implement training_epoch_end in the LightningModule and access outputs via the module OR

2. Cache data across train batch hooks inside the callback implementation to post-process in this hook.

on_validation_epoch_end(trainer: Trainer, pl_module: LightningModule) → None

Called when the val epoch ends.

class cca_zoo.deepmodels._callbacks.GenerativeCallback

Bases: Callback

on_validation_epoch_end(trainer: Trainer, pl_module: LightningModule) → None

Called when the val epoch ends.

Model Architectures

class cca_zoo.deepmodels.architectures.Encoder(latent_dims: int, variational: bool = False, feature_size: int = 784, layer_sizes: Optional[tuple] = None, activation=LeakyReLU(negative_slope=0.01), dropout=0)

Bases: _BaseEncoder

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class cca_zoo.deepmodels.architectures.Decoder(latent_dims: int, feature_size: int = 784, layer_sizes: Optional[tuple] = None, activation=LeakyReLU(negative_slope=0.01), dropout=0)

Bases: _BaseDecoder

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class cca_zoo.deepmodels.architectures.CNNEncoder(latent_dims: int, variational: bool = False, feature_size: Iterable = (28, 28), channels: Optional[tuple] = None, kernel_sizes: Optional[tuple] = None, stride: Optional[tuple] = None, padding: Optional[tuple] = None, activation=LeakyReLU(negative_slope=0.01), dropout=0)

Bases: _BaseEncoder

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class cca_zoo.deepmodels.architectures.CNNDecoder(latent_dims: int, feature_size: Iterable = (28, 28), channels: Optional[tuple] = None, kernel_sizes=None, strides=None, paddings=None, activation=LeakyReLU(negative_slope=0.01), dropout=0)

Bases: _BaseDecoder

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class cca_zoo.deepmodels.architectures.LinearEncoder(latent_dims: int, feature_size: int, variational: bool = False)

Bases: _BaseEncoder

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class cca_zoo.deepmodels.architectures.LinearDecoder(latent_dims: int, feature_size: int)

Bases: _BaseDecoder

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.