hunk.code_window:         Args:
               img_size (int, tuple): input image size
               patch_size (int, tuple): patch size
               in_chans (int): number of input channels
               num_classes (int): number of classes for classification head
               global_pool (str): type of global pooling for final sequence (default: 'token')
               embed_dim (int): embedding dimension
               depth (int): depth of transformer
               num_heads (int): number of attention heads
  
hunk.labels: keep keep keep keep keep replace keep keep keep
hunk.after_edit:             global_pool (str): type of global pooling for final sequence (default: 'avg')
  
hunk.file_path: timm/models/vision_transformer_relpos.py
hunk.type: replace
file: # SWSL ResNeXt
  
  A **ResNeXt** repeats a [building block](https://paperswithcode.com/method/resnext-block) that aggregates a set of transformations with the same topology. Compared to a [ResNet](https://paperswithcode.com/method/resnet), it exposes a new dimension,  *cardinality* (the size of the set of transformations) $C$, as an essential factor in addition to the dimensions of depth and width. 
  
  The models in this collection utilise semi-weakly supervised learning to improve the performance of the model. The approach brings important gains to standard architectures for image, video and fine-grained classification. 
  
  Please note the CC-BY-NC 4.0 license on theses weights, non-commercial use only.
  
  {% include 'code_snippets.md' %}
  
  ## How do I train this model?
  
  You can follow the [timm recipe scripts](https://rwightman.github.io/pytorch-image-models/scripts/) for training a new model afresh.
  
  ## Citation
  
  ```BibTeX
  @article{DBLP:journals/corr/abs-1905-00546,
    author    = {I. Zeki Yalniz and
                 Herv{\'{e}} J{\'{e}}gou and
                 Kan Chen and
                 Manohar Paluri and
                 Dhruv Mahajan},
    title     = {Billion-scale semi-supervised learning for image classification},
    journal   = {CoRR},
    volume    = {abs/1905.00546},
    year      = {2019},
    url       = {http://arxiv.org/abs/1905.00546},
    archivePrefix = {arXiv},
    eprint    = {1905.00546},
    timestamp = {Mon, 28 Sep 2020 08:19:37 +0200},
    biburl    = {https://dblp.org/rec/journals/corr/abs-1905-00546.bib},
    bibsource = {dblp computer science bibliography, https://dblp.org}
  }
  ```
  
  <!--
  Type: model-index
  Collections:
  - Name: SWSL ResNext
    Paper:
      Title: Billion-scale semi-supervised learning for image classification
      URL: https://paperswithcode.com/paper/billion-scale-semi-supervised-learning-for
  Models:
  - Name: swsl_resnext101_32x16d
    In Collection: SWSL ResNext
    Metadata:
      FLOPs: 46623691776
      Parameters: 194030000
      File Size: 777518664
      Architecture:
      - 1x1 Convolution
      - Batch Normalization
      - Convolution
      - Global Average Pooling
      - Grouped Convolution
      - Max Pooling
      - ReLU
      - ResNeXt Block
      - Residual Connection
      - Softmax
      Tasks:
      - Image Classification
      Training Techniques:
      - SGD with Momentum
      - Weight Decay
      Training Data:
      - IG-1B-Targeted
      - ImageNet
      Training Resources: 64x GPUs
      ID: swsl_resnext101_32x16d
      LR: 0.0015
      Epochs: 30
      Layers: 101
      Crop Pct: '0.875'
      Batch Size: 1536
      Image Size: '224'
      Weight Decay: 0.0001
      Interpolation: bilinear
    Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/resnet.py#L1009
    Weights: https://dl.fbaipublicfiles.com/semiweaksupervision/model_files/semi_weakly_supervised_resnext101_32x16-f3559a9c.pth
    Results:
    - Task: Image Classification
      Dataset: ImageNet
      Metrics:
        Top 1 Accuracy: 83.34%
        Top 5 Accuracy: 96.84%
  - Name: swsl_resnext101_32x4d
    In Collection: SWSL ResNext
    Metadata:
      FLOPs: 10298145792
      Parameters: 44180000
      File Size: 177341913
      Architecture:
      - 1x1 Convolution
      - Batch Normalization
      - Convolution
      - Global Average Pooling
      - Grouped Convolution
      - Max Pooling
      - ReLU
      - ResNeXt Block
      - Residual Connection
      - Softmax
      Tasks:
      - Image Classification
      Training Techniques:
      - SGD with Momentum
      - Weight Decay
      Training Data:
      - IG-1B-Targeted
      - ImageNet
      Training Resources: 64x GPUs
      ID: swsl_resnext101_32x4d
      LR: 0.0015
      Epochs: 30
      Layers: 101
      Crop Pct: '0.875'
      Batch Size: 1536
      Image Size: '224'
      Weight Decay: 0.0001
      Interpolation: bilinear
    Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/resnet.py#L987
    Weights: https://dl.fbaipublicfiles.com/semiweaksupervision/model_files/semi_weakly_supervised_resnext101_32x4-3f87e46b.pth
    Results:
    - Task: Image Classification
      Dataset: ImageNet
      Metrics:
        Top 1 Accuracy: 83.22%
        Top 5 Accuracy: 96.77%
  - Name: swsl_resnext101_32x8d
    In Collection: SWSL ResNext
    Metadata:
      FLOPs: 21180417024
      Parameters: 88790000
      File Size: 356056638
      Architecture:
      - 1x1 Convolution
      - Batch Normalization
      - Convolution
      - Global Average Pooling
      - Grouped Convolution
      - Max Pooling
      - ReLU
      - ResNeXt Block
      - Residual Connection
      - Softmax
      Tasks:
      - Image Classification
      Training Techniques:
      - SGD with Momentum
      - Weight Decay
      Training Data:
      - IG-1B-Targeted
      - ImageNet
      Training Resources: 64x GPUs
      ID: swsl_resnext101_32x8d
      LR: 0.0015
      Epochs: 30
      Layers: 101
      Crop Pct: '0.875'
      Batch Size: 1536
      Image Size: '224'
      Weight Decay: 0.0001
      Interpolation: bilinear
    Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/resnet.py#L998
    Weights: https://dl.fbaipublicfiles.com/semiweaksupervision/model_files/semi_weakly_supervised_resnext101_32x8-b4712904.pth
    Results:
    - Task: Image Classification
      Dataset: ImageNet
      Metrics:
        Top 1 Accuracy: 84.27%
        Top 5 Accuracy: 97.17%
  - Name: swsl_resnext50_32x4d
    In Collection: SWSL ResNext
    Metadata:
      FLOPs: 5472648192
      Parameters: 25030000
      File Size: 100428550
      Architecture:
      - 1x1 Convolution
      - Batch Normalization
      - Convolution
      - Global Average Pooling
      - Grouped Convolution
      - Max Pooling
      - ReLU
      - ResNeXt Block
      - Residual Connection
      - Softmax
      Tasks:
      - Image Classification
      Training Techniques:
      - SGD with Momentum
      - Weight Decay
      Training Data:
      - IG-1B-Targeted
      - ImageNet
      Training Resources: 64x GPUs
      ID: swsl_resnext50_32x4d
      LR: 0.0015
      Epochs: 30
      Layers: 50
      Crop Pct: '0.875'
      Batch Size: 1536
      Image Size: '224'
      Weight Decay: 0.0001
      Interpolation: bilinear
    Code: https://github.com/rwightman/pytorch-image-models/blob/9a25fdf3ad0414b4d66da443fe60ae0aa14edc84/timm/models/resnet.py#L976
    Weights: https://dl.fbaipublicfiles.com/semiweaksupervision/model_files/semi_weakly_supervised_resnext50_32x4-72679e44.pth
    Results:
    - Task: Image Classification
      Dataset: ImageNet
      Metrics:
        Top 1 Accuracy: 82.17%
        Top 5 Accuracy: 96.23%
  -->
  