ModelZoo

FederatedScope provides many built-in models in different deep learning fields, including Computer Vision, Natural Language Processing, Graph, Recommendation Systems, and Speech. Furthermore, more models are on the way!

To use our ModelZoo, set cfg.model.type = Model_NAME. And you can configure the model-related hyperparameters via ayamlfile.

# Some methods may leverage more than one model in each trainer
cfg.model.model_num_per_trainer = 1 
# Model name
cfg.model.type = 'lr'
cfg.model.use_bias = True
# For graph model
cfg.model.task = 'node'
# Hidden dim
cfg.model.hidden = 256
# Drop out ratio
cfg.model.dropout = 0.5
# in_channels dim. If 0, model will be built by data.shape
cfg.model.in_channels = 0
# out_channels dim. If 0, model will be built by label.shape
cfg.model.out_channels = 1
# In GPR-GNN, K = gnn_layer
cfg.model.gnn_layer = 2
cfg.model.graph_pooling = 'mean'
cfg.model.embed_size = 8
cfg.model.num_item = 0
cfg.model.num_user = 0

For more model-related settings, please refer to each model.

Computer Vision

  • ConvNet2
    ConvNet2 (from federatedscope/cv/model) is a two-layer CNN for image classification. (cfg.model.type = 'convnet2') 
    class ConvNet2(Module):
 def __init__(self, in_channels, h=32, w=32, hidden=2048, class_num=10, use_bn=True):
     ...
  • ConvNet5
    ConvNet5 (from federatedscope/cv/model) is a five-layer CNN for image classification. (cfg.model.type = 'convnet5') 
    class ConvNet5(Module):
 def __init__(self, in_channels, h=32, w=32, hidden=2048, class_num=10):
     ...
  • VGG11
    VGG11 [1] (from federatedscope/cv/model) is an 11 layer CNN with very small (3x3) convolution filters for image classification. It is from Very Deep Convolutional Networks for Large-Scale Image Recognition. (cfg.model.type = 'vgg11') 
    class VGG11(Module):
 def __init__(self, in_channels, h=32, w=32, hidden=128, class_num=10):
     ...

Natural Language Processing

  • LSTM
    LSTM [2] (from federatedscope/nlp/model) is a type of RNN that solves the vanishing gradient problem through additional cells, input and output gates. (cfg.model.type = 'lstm') 
    class LSTM(nn.Module):
 def __init__(self, in_channels, hidden, out_channels, n_layers=2, embed_size=8):
     ...

Graph

  • GCN
    GCN [3] (from federatedscope/gfl/model) is a kind of Graph Neural Networks from Semi-supervised Classification with Graph Convolutional Networks, which is adapted for node-level, link-level and graph-level tasks. (cfg.model.type = 'gcn', cfg.model.task = 'node') 
    class GCN_Net(torch.nn.Module):
 def __init__(self, in_channels, out_channels, hidden=64, max_depth=2, dropout=.0):
     ...
  • GAT
    GAT [4] (from federatedscope/gfl/model) is a kind of Graph Neural Networks from Graph Attention Networks. GAT employ attention mechanisms to node neighbors to learn attention coefficients, which is adapted for node-level, link-level and graph-level tasks.  (cfg.model.type = 'gat', cfg.model.task = 'node' # node, link or graph) 
    class GAT_Net(torch.nn.Module):
 def __init__(self, in_channels, out_channels, hidden=64, max_depth=2, dropout=.0):
     ...
  • GraphSAGE
    GraphSAGE [5] (from federatedscope/gfl/model) is a general inductive GNN framework, from Inductive Representation Learning on Large Graphs. GraphSAGE learns a function that generates embeddings by sampling and aggregating from the local neighborhood of each node, which is adapted for node-level and link-level tasks.  (cfg.model.type = 'sage', cfg.model.task = 'node' # node, link or graph) 
    class SAGE_Net(torch.nn.Module):
 def __init__(self, in_channels, out_channels, hidden=64, max_depth=2, dropout=.0):
     ...
  • GPR-GNN
    GPR-GNN [6] (from federatedscope/gfl/model) adaptively learns the Generalized PageRank weights so as to jointly optimize node feature and topological information extraction from Adaptive Universal Generalized PageRank Graph Neural Network, which is adapted for node-level and link-level tasks.  (cfg.model.type = 'gpr', cfg.model.task = 'node' # node or link) 
    class GPR_Net(torch.nn.Module):
 def __init__(self, in_channels, out_channels, hidden=64, K=10, dropout=.0, ppnp='GPR_prop', alpha=0.1, Init='PPR'):
     ...
  • GIN
    GIN [7] (from federatedscope/gfl/model) generalizes the Weisfeiler-Lehman test and achieves maximum discriminative power among GNNs from How Powerful are Graph Neural Networks? which is adapted for graph-level tasks.  (cfg.model.type = 'gin', cfg.model.task = 'graph') 
    class GIN_Net(torch.nn.Module):
 def __init__(self, in_channels, out_channels, hidden=64, max_depth=2, dropout=.0):
     ...

Recommendation System

  • MF models
    MF model [8] (from federatedscope/mf/model) has two trainable parameters: user embedding and item embedding. Based on the given federated setting, they share different embedding with the other participators. FederatedScope achieves VMFNetand HMFNetto support federated MF, and both of them inherit the basic MF model class BasicMFNet.
class VMFNet(BasicMFNet):
    name_reserve = "embed_item"


class HMFNet(BasicMFNet):
    name_reserve = "embed_user"

Speech

Coming Soon!

References

[1] Simonyan, Karen, and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv 2014.

[2] Hochreiter, Sepp, and Jürgen Schmidhuber. “Long short-term memory.” Neural computation 1997.

[3] Kipf, Thomas N., and Max Welling. “Semi-supervised classification with graph convolutional networks.” arXiv 2016.

[4] Veličković, Petar, et al. “Graph attention networks.” ICLR 2018.

[5] Hamilton, Will, Zhitao Ying, and Jure Leskovec. “Inductive representation learning on large graphs.” NeurIPS 2017.

[6] Chien, Eli, et al. “Adaptive universal generalized pagerank graph neural network.” ICLR 2021.

[7] Xu, Keyulu, et al. “How powerful are graph neural networks?.”  ICLR 2019.

[8] Yongjie, Du, et al. “Federated matrix factorization for privacy-preserving recommender systems” VLDB 2022.

Updated: