Deep Models

Models

class cca_zoo.deepmodels.DCCA(latent_dims: int, objective=<class 'cca_zoo.deepmodels.utils.objectives.MCCA'>, encoders=None, r: float = 0, eps: float = 1e-05, **kwargs)

Bases: _BaseDeep, _BaseCCA

A class used to fit a DCCA model.

References

Andrew, Galen, et al. “Deep canonical correlation analysis.” International conference on machine learning. PMLR, 2013.

forward(views, **kwargs)

We use the forward model to define the transformation of views to the latent space :param views: batches for each view separated by commas

loss(views, **kwargs)

Loss for model :param views: :param args: :param kwargs: :return:

pairwise_correlations(loader: DataLoader, train=False)

Calculates correlation for entire batch from dataloader

Parameters

• loader – a dataloader that matches the structure of that used for training

• train – whether to fit final linear transformation

Returns

by default returns the average pairwise correlation in each dimension (for 2 views just the correlation)

score(loader: DataLoader, **kwargs)

Returns average correlation in each dimension (averages over all pairs for multiview)

Parameters

• **kwargs –

• loader – a dataloader that matches the structure of that used for training

• train – whether to fit final linear transformation

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

class cca_zoo.deepmodels.DCCAE(latent_dims: int, objective=<class 'cca_zoo.deepmodels.utils.objectives.MCCA'>, encoders=None, decoders=None, r: float = 0, eps: float = 1e-05, lam=0.5, latent_dropout=0, img_dim=None, recon_loss_type='mse', **kwargs)

Bases: DCCA, _GenerativeMixin

A class used to fit a DCCAE model.

References

Wang, Weiran, et al. “On deep multi-view representation learning.” International conference on machine learning. PMLR, 2015.

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Loss for model :param views: :param args: :param kwargs: :return:

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

class cca_zoo.deepmodels.DCCA_NOI(latent_dims: int, N: int, encoders=None, r: float = 0, rho: float = 0.2, eps: float = 1e-09, shared_target: bool = False, **kwargs)

Bases: DCCA

A class used to fit a DCCA model by non-linear orthogonal iterations

References

Wang, Weiran, et al. “Stochastic optimization for deep CCA via nonlinear orthogonal iterations.” 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 2015.

loss(views, **kwargs)

Loss for model :param views: :param args: :param kwargs: :return:

class cca_zoo.deepmodels.DCCA_SDL(latent_dims: int, N: int, encoders=None, r: float = 0, rho: float = 0.2, eps: float = 1e-05, shared_target: bool = False, lam=0.5, **kwargs)

Bases: DCCA_NOI

A class used to fit a Deep CCA by Stochastic Decorrelation model.

References

Chang, Xiaobin, Tao Xiang, and Timothy M. Hospedales. “Scalable and effective deep CCA via soft decorrelation.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

forward(views, **kwargs)

We use the forward model to define the transformation of views to the latent space :param views: batches for each view separated by commas

loss(views, **kwargs)

Loss for model :param views: :param args: :param kwargs: :return:

class cca_zoo.deepmodels.DVCCA(latent_dims: int, encoders=None, decoders=None, private_encoders: Optional[Iterable[_BaseEncoder]] = None, latent_dropout=0, img_dim=None, recon_loss_type='mse', **kwargs)

Bases: _BaseDeep, _GenerativeMixin

A class used to fit a DVCCA model.

References

Wang, Weiran, et al. ‘Deep variational canonical correlation analysis.’ arXiv preprint arXiv:1610.03454 (2016). https: // arxiv.org / pdf / 1610.03454.pdf https: // github.com / pytorch / examples / blob / master / vae / main.py

forward(views, mle=True, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Loss for model :param views: :param args: :param kwargs: :return:

transform(loader: DataLoader)

Parameters

loader – a dataloader that matches the structure of that used for training

Returns

transformed views

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

class cca_zoo.deepmodels.BarlowTwins(latent_dims: int, encoders=None, lam=1, **kwargs)

Bases: DCCA

A class used to fit a Barlow Twins model.

References

Zbontar, Jure, et al. “Barlow twins: Self-supervised learning via redundancy reduction.” arXiv preprint arXiv:2103.03230 (2021).

forward(views, **kwargs)

We use the forward model to define the transformation of views to the latent space :param views: batches for each view separated by commas

loss(views, **kwargs)

Loss for model :param views: :param args: :param kwargs: :return:

class cca_zoo.deepmodels.DTCCA(latent_dims: int, encoders=None, r: float = 0, eps: float = 1e-05, **kwargs)

Bases: DCCA

A class used to fit a DTCCA model.

Is just a thin wrapper round DCCA with the DTCCA objective and a TCCA post-processing

References

Wong, Hok Shing, et al. “Deep Tensor CCA for Multi-view Learning.” IEEE Transactions on Big Data (2021).

class cca_zoo.deepmodels.SplitAE(latent_dims: int, encoder: ~cca_zoo.deepmodels.utils.architectures._BaseEncoder = <class 'cca_zoo.deepmodels.utils.architectures.Encoder'>, decoders=None, latent_dropout=0, recon_loss_type='mse', img_dim=None, **kwargs)

Bases: _BaseDeep, _GenerativeMixin

A class used to fit a Split Autoencoder model.

References

Ngiam, Jiquan, et al. “Multimodal deep learning.” ICML. 2011.

Parameters

• latent_dims – # latent dimensions

• encoder – list of encoder networks

• decoders – list of decoder networks

forward(views, **kwargs)

Forward method for the model. Outputs latent encoding for each view

Parameters

• views –

• kwargs –

Returns

loss(views, **kwargs)

Loss for model :param views: :param args: :param kwargs: :return:

configure_callbacks()

Configure model-specific callbacks. When the model gets attached, e.g., when .fit() or .test() gets called, the list or a callback returned here will be merged with the list of callbacks passed to the Trainer’s callbacks argument. If a callback returned here has the same type as one or several callbacks already present in the Trainer’s callbacks list, it will take priority and replace them. In addition, Lightning will make sure ModelCheckpoint callbacks run last.

Returns

A callback or a list of callbacks which will extend the list of callbacks in the Trainer.

Example:

def configure_callbacks(self):
    early_stop = EarlyStopping(monitor="val_acc", mode="max")
    checkpoint = ModelCheckpoint(monitor="val_loss")
    return [early_stop, checkpoint]

Note

Certain callback methods like on_init_start() will never be invoked on the new callbacks returned here.

cca_zoo.deepmodels.get_dataloaders(dataset, val_dataset=None, batch_size=None, val_batch_size=None, drop_last=True, val_drop_last=False, shuffle_train=False, pin_memory=True, num_workers=0, persistent_workers=True)

A utility function to allow users to quickly get hold of the dataloaders required by pytorch lightning

Parameters

• dataset – A CCA dataset used for training

• val_dataset – An optional CCA dataset used for validation

• batch_size – batch size of train loader

• val_batch_size – batch size of val loader

• num_workers – number of workers used

• pin_memory – pin memory used by pytorch - True tends to speed up training

• shuffle_train – whether to shuffle training data

• val_drop_last – whether to drop the last incomplete batch from the validation data

• drop_last – whether to drop the last incomplete batch from the train data

• persistent_workers – whether to keep workers alive after dataloader is destroyed

Deep Objectives

Callbacks

Model Architectures