Upload README.md

README.md

@@ -1,60 +1,67 @@
 # SPG: Sequential Policy Gradient for Adaptive Hyperparameter Optimization
 
 
-## Model zoo
-We provide baseline models and SPG-trained models, all available for download at the following links:
+## Model Zoo: Adaptive Hyperparameter Optimization (HPO) via SPG Algorithm
 
 
-`Table 1: Model comparison on the ImageNet-1K dataset.`
-| Model | SPG  | # Params | Acc@1 (%) | Acc@5 (%) | Weights |
-|-------|------|----------|-----------|-----------|---------|
-| MobileNet-V2 | ❌ | 3.5 M | 71.878 | 90.286 | <a href='https://download.pytorch.org/models/mobilenet_v2-b0353104.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
-| MobileNet-V2 | ✅ | 3.5 M | 72.104 | 90.316 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/mobilenet_v2/model_32.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> |
-| ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 | <a href='https://download.pytorch.org/models/resnet50-0676ba61.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
+`Table 1: Performance of pre-trained vs. SPG-retrained models on ImageNet-1K`
+| Model | SPG  | # Params | Acc@1 (%) | Acc@5 (%) | Weights | Command to reproduce |
+|-------|------|----------|-----------|-----------|---------|----------------------|
+| MobileNet-V2 | ❌ | 3.5 M | 71.878 | 90.286 | <a href='https://download.pytorch.org/models/mobilenet_v2-b0353104.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#mobilenetv2'>Recipe</a> |
+| MobileNet-V2 | ✅ | 3.5 M | 72.104 | 90.316 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/mobilenet_v2/model_32.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | [examples/image-classification/run.sh](#-Retrain-model-on-ImageNet-1K) |
+| ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 | <a href='https://download.pytorch.org/models/resnet50-0676ba61.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> |
 | ResNet-50 | ✅ | 25.6 M | 77.234 | 93.322 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/resnet50/model_35.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> |
-| EfficientNet-V2-M | ❌ | 54.1 M | 85.112 | 97.156 | <a href='https://download.pytorch.org/models/efficientnet_v2_m-dc08266a.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
+| EfficientNet-V2-M | ❌ | 54.1 M | 85.112 | 97.156 | <a href='https://download.pytorch.org/models/efficientnet_v2_m-dc08266a.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#efficientnet-v2'>Recipe</a> |
 | EfficientNet-V2-M | ✅ | 54.1 M | 85.218 | 97.208 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/efficientnet_v2_m/model_7.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> |
-| ViT-B16 | ❌ | 86.6 M | 81.072 | 95.318 | <a href='https://download.pytorch.org/models/vit_b_16-c867db91.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
+| ViT-B16 | ❌ | 86.6 M | 81.072 | 95.318 | <a href='https://download.pytorch.org/models/vit_b_16-c867db91.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#vit_b_16'>Recipe</a> |
 | ViT-B16 | ✅ | 86.6 M | 81.092 | 95.304 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/vit_b_16/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> |
 
 
-`Table 2: All models are evaluated a subset of COCO val2017, on the 21 categories (including
-"background") that are present in the Pascal VOC dataset.`
 
-`All model reported on TorchVision (with weight COCO_WITH_VOC_LABELS_V1) were benchmarked using only 20 categories. Researchers should first download the pre-trained model from TorchVision and conduct re-evaluation under the 21-category framework.`
+`Table 2: Performance of pre-trained vs. SPG-retrained models. All models are evaluated a subset of COCO val2017, on the 21 categories (including "background") that are present in the Pascal VOC dataset.`
 
-| Model               | SPG | # Params | mIoU (%)   | pixelwise Acc (%)   | Weights |
-|---------------------|-----|----------|------------|---------------------|---------|
-| FCN-ResNet50        | ❌  | 35.3 M | 58.9 | 90.9 | <a href='https://download.pytorch.org/models/fcn_resnet50_coco-1167a1af.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
+⚠️`All model reported on TorchVision (with weight COCO_WITH_VOC_LABELS_V1) were benchmarked using only 20 categories. Researchers should first download the pre-trained model from TorchVision and conduct re-evaluation under the 21-category framework.`
+
+| Model               | SPG | # Params | mIoU (%)   | pixelwise Acc (%)   | Weights | Command to reproduce |
+|---------------------|-----|----------|------------|---------------------|---------|----------------------|
+| FCN-ResNet50        | ❌  | 35.3 M | 58.9 | 90.9 | <a href='https://download.pytorch.org/models/fcn_resnet50_coco-1167a1af.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#fcn_resnet50'>Recipe</a> |
 | FCN-ResNet50        | ✅  | 35.3 M | 59.4 | 90.9 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/fcn_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> |
-| FCN-ResNet101       | ❌  | 54.3 M | 62.2 | 91.1 | <a href='https://download.pytorch.org/models/fcn_resnet101_coco-7ecb50ca.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
+| FCN-ResNet101       | ❌  | 54.3 M | 62.2 | 91.1 | <a href='https://download.pytorch.org/models/fcn_resnet101_coco-7ecb50ca.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
 | FCN-ResNet101       | ✅  | 54.3 M | 62.4 | 91.1 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/fcn_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> |
-| DeepLabV3-ResNet50  | ❌  | 42.0 M | 63.8 | 91.5 | <a href='https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
+| DeepLabV3-ResNet50  | ❌  | 42.0 M | 63.8 | 91.5 | <a href='https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet50'>Recipe</a> |
 | DeepLabV3-ResNet50  | ✅  | 42.0 M | 64.2 | 91.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/deeplabv3_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> |
-| DeepLabV3-ResNet101 | ❌  | 61.0 M | 65.3 | 91.7 | <a href='https://download.pytorch.org/models/deeplabv3_resnet101_coco-586e9e4e.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> |
+| DeepLabV3-ResNet101 | ❌  | 61.0 M | 65.3 | 91.7 | <a href='https://download.pytorch.org/models/deeplabv3_resnet101_coco-586e9e4e.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
 | DeepLabV3-ResNet101 | ✅  | 61.0 M | 65.7 | 91.8 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/deeplabv3_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> |
 
 
-`Table 3: Performance of models for transfer learning trained with fine-tuning (FT) vs. SPG.`
-| Task  | SPG  | Metric Type       | Performance (%) | Weights |
-|-------|------|------------------|-----------------|---------|
-| CoLA  | ❌   | Matthews coor     | 56.53           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> |
+
+`Table X: Performance comparison of fine-tuned vs. SPG-retrained models across NLP and speech benchmarks.`
+- GLUE (Text classification: BERT on CoLA, SST-2, MRPC, QQP, QNLI, and RTE task)
+- SQuAD (Question answering: BERT)
+- SUPERB (Speech classification: Wav2Vec2 for Audio Classification (AC))
+
+| Task  | SPG  | Metric Type       | Performance (%) | Weights | Command to reproduce |
+|-------|------|-------------------|-----------------|---------|----------------------|
+| CoLA  | ❌   | Matthews coor     | 56.53           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
 | CoLA  | ✅   | Matthews coor     | 62.13           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> |
-| SST-2 | ❌   | Accuracy         | 92.32           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> |
+| SST-2 | ❌   | Accuracy         | 92.32           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
 | SST-2 | ✅   | Accuracy         | 92.54           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> |
-| MRPC  | ❌   | F1/Accuracy      | 88.85/84.09     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> |
+| MRPC  | ❌   | F1/Accuracy      | 88.85/84.09     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
 | MRPC  | ✅   | F1/Accuracy      | 91.10/87.25     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> |
-| QQP   | ❌   | F1/Accuracy      | 87.49/90.71     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> |
+| QQP   | ❌   | F1/Accuracy      | 87.49/90.71     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
 | QQP   | ✅   | F1/Accuracy      | 89.72/90.88     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> |
-| QNLI  | ❌   | Accuracy         | 90.66           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> |
+| QNLI  | ❌   | Accuracy         | 90.66           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
 | QNLI  | ✅   | Accuracy         | 91.10           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> |
-| RTE   | ❌   | Accuracy         | 65.70           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> |
+| RTE   | ❌   | Accuracy         | 65.70           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
 | RTE   | ✅   | Accuracy         | 72.56           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> |
-| Q/A*  | ❌   | F1/Extra match   | 88.52/81.22     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-question_answering-yellow'></a> |
+| Q/A*  | ❌   | F1/Extra match   | 88.52/81.22     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-question_answering-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering#fine-tuning-bert-on-squad10'>Recipe</a> |
 | Q/A*  | ✅   | F1/Extra match   | 88.67/81.51     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> |
-| AC†   | ❌   | Accuracy         | 98.26           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-audio_classification-yellow'></a> |
+| AC†   | ❌   | Accuracy         | 98.26           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-audio_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification#single-gpu'>Recipe</a> |
 | AC†   | ✅   | Accuracy         | 98.31           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> |
 
+
+## Model Zoo: Neural Architecture Search (NAS) via SPG Algorithm
+
 ## Requirements
 
 1. Install `torch>=2.0.0+cu118`.
@@ -90,11 +97,12 @@ We provide baseline models and SPG-trained models, all available for download at
 
 ## Training
 
-### Model retraining
-We utilize recipes similar to those in [PyTorch Vision's classification reference](https://github.com/pytorch/vision/blob/main/references/classification/README.md) to retrain MobileNet-V2, ResNet, EfficientNet-V2, and ViT using our SPG on ImageNet. You can run the following command:
+<a id="#-Retrain-model-on-ImageNet-1K"></a>
+### Retrain model on ImageNet-1K
+We use training recipes similar to those in [PyTorch Vision's classification reference](https://github.com/pytorch/vision/blob/main/references/classification/README.md) to retrain MobileNet-V2, ResNet, EfficientNet-V2, and ViT with our SPG on ImageNet-1K. The following command can be used:
 
-```train
-cd image-classification
+```bash
+cd ./examples/image-classification
 
 # MobileNet-V2
 torchrun --nproc_per_node=4 train.py\
@@ -129,10 +137,12 @@ torchrun --nproc_per_node=4 train.py\
   --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 --print-freq 100
 ```
 
-We utilize recipes similar to those in [PyTorch Vision's segmentation reference](https://github.com/pytorch/vision/blob/main/references/segmentation/README.md) to retrain FCN and DeepLab-V3 using our SPG on COCO dataset. You can run the following command:
+### Retrain model on MS-COCO 2017
+We use training recipes similar to those in [PyTorch Vision's segmentation reference](https://github.com/pytorch/vision/blob/main/references/segmentation/README.md) to retrain FCN and DeepLab-V3 with our SPG on COCO dataset. The following command can be used:
 
-```train
-cd semantic-segmentation
+```bash
+
+cd ./examples/semantic-segmentation
 
 # FCN-ResNet50
 torchrun --nproc_per_node=4 train.py\
@@ -166,100 +176,20 @@ torchrun --nproc_per_node=4 train.py\
   --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \
   --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
 ```
+</details>
 
-### Transfer learning
-We utilize recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain BERT and Wav2Vec using our SPG on GLUE benchmark, SquAD dataset, and SUPERB benchmark. You can run the following command:
-
-```train
-cd text-classification
-
-# Task: CoLA 
-CUDA_VISIBLE_DEVICES=0 python run_glue.py \
-  --model_name_or_path google-bert/bert-base-cased \
-  --task_name "cola" \
-  --do_train \
-  --do_eval \
-  --max_seq_length 128 \
-  --per_device_train_batch_size 32 \
-  --learning_rate 2.5e-5 \
-  --num_train_epochs 6 \
-  --output_dir "cola" \
-  --overwrite_output_dir \
-  --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
-
-# Task: SST-2
-CUDA_VISIBLE_DEVICES=0 python run_glue.py \
-  --model_name_or_path google-bert/bert-base-cased \
-  --task_name "sst2" \
-  --do_train \
-  --do_eval \
-  --max_seq_length 128 \
-  --per_device_train_batch_size 64 \
-  --learning_rate 3e-5 \
-  --num_train_epochs 5 \
-  --output_dir "sst2" \
-  --overwrite_output_dir \
-  --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
-
-# Task: MRPC
-CUDA_VISIBLE_DEVICES=0 python run_glue.py \
-  --model_name_or_path google-bert/bert-base-cased \
-  --task_name "mrpc" \
-  --do_train \
-  --do_eval \
-  --max_seq_length 128 \
-  --per_device_train_batch_size 16 \
-  --learning_rate 2e-5 \
-  --num_train_epochs 4 \
-  --output_dir "mrpc" \
-  --overwrite_output_dir \
-  --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
+### Transfer learning on GLUE
+We use recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain BERT with our SPG on GLUE benchmark. The following command can be used:
 
-# Task: QQP
-CUDA_VISIBLE_DEVICES=0 python run_glue.py \
-  --model_name_or_path google-bert/bert-base-cased \
-  --task_name "qqp" \
-  --do_train \
-  --do_eval \
-  --max_seq_length 128 \
-  --per_device_train_batch_size 32 \
-  --learning_rate 1e-5 \
-  --num_train_epochs 10 \
-  --output_dir "qqp" \
-  --overwrite_output_dir \
-  --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
-
-# Task: QNLI
-CUDA_VISIBLE_DEVICES=0 python run_glue.py \
-  --model_name_or_path google-bert/bert-base-cased \
-  --task_name "qnli" \
-  --do_train \
-  --do_eval \
-  --max_seq_length 128 \
-  --per_device_train_batch_size 32 \
-  --learning_rate 2e-5 \
-  --num_train_epochs 10 \
-  --output_dir "qnli" \
-  --overwrite_output_dir \
-  --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
+```bash
+cd ./examples/text-classification && bash run.sh
+```
 
-# Task: RTE
-CUDA_VISIBLE_DEVICES=0 python run_glue.py \
-  --model_name_or_path google-bert/bert-base-cased \
-  --task_name "rte" \
-  --do_train \
-  --do_eval \
-  --max_seq_length 128 \
-  --per_device_train_batch_size 32 \
-  --learning_rate 5e-5 \
-  --num_train_epochs 5 \
-  --output_dir "rte" \
-  --overwrite_output_dir \
-  --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
-    
+### Transfer learning on SQuAD
+We use recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain Wav2Vec with our SPG on SQuAD dataset. The following command can be used:
 
-# Task: audio classification
-cd ../audio-classification
+```bash
+cd ./examples/audio-classification
 CUDA_VISIBLE_DEVICES=0 python run_audio_classification.py \
   --model_name_or_path facebook/wav2vec2-base \
   --dataset_name superb \
@@ -290,10 +220,14 @@ CUDA_VISIBLE_DEVICES=0 python run_audio_classification.py \
   --seed 0 \
   --push_to_hub \
   --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
+```
 
 
-# Task: question answering
-cd ../question-answering
+### Transfer learning on SUPERB
+We use recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain BERT with our SPG on SUPERB benchmark. The following command can be used:
+
+```bash
+cd ./examples/question-answering
 CUDA_VISIBLE_DEVICES=0 python run_qa.py \
   --model_name_or_path google-bert/bert-base-uncased \
   --dataset_name squad \
@@ -309,12 +243,13 @@ CUDA_VISIBLE_DEVICES=0 python run_qa.py \
   --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
 ```
 
-### Network Architecture Search
-We conduct Neural Architecture Search (NAS) on the ResNet architecture using the ImageNet dataset. You can run the following command:
+### Neural Architecture Search for ResNet on ImageNet-1K
+We conduct Neural Architecture Search (NAS) for the ResNet architecture on the ImageNet dataset. The following command can be used:
 
-```train
-cd ../neural-architecture-search
+```bash
+cd ./examples/neural-architecture-search
 
+# During Neural Architecture Search (NAS), we explore ResNet-18, ResNet-27, ResNet-36, and ResNet-45. After retraining with SPG algorithm, we retain only ResNet-18 and discard the others.
 torchrun --nproc_per_node=4 train.py\
     --data-path /home/cs/Documents/datasets/imagenet\
     --model resnet18 --output-dir resnet18 --weights ResNet18_Weights.IMAGENET1K_V1\
@@ -322,7 +257,7 @@ torchrun --nproc_per_node=4 train.py\
     --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\
     --apply-trp --trp-depths 3 3 3 --trp-planes 256 --trp-lambdas 0.4 0.2 0.1 --print-freq 100
 
-# ✅ Test:  Acc@1 73.900 Acc@5 91.536
+# During Neural Architecture Search (NAS), we explore ResNet-34, ResNet-40, ResNet-46, and ResNet-52. After retraining with SPG algorithm, we retain only ResNet-34 and discard the others.
 torchrun --nproc_per_node=4 train.py\
     --data-path /home/cs/Documents/datasets/imagenet\
     --model resnet34 --output-dir resnet34 --weights ResNet34_Weights.IMAGENET1K_V1\
@@ -330,8 +265,7 @@ torchrun --nproc_per_node=4 train.py\
     --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\
     --apply-trp --trp-depths 2 2 2 --trp-planes 256 --trp-lambdas 0.4 0.2 0.1 --print-freq 100
 
-
-# ✅ Test:  Acc@1 76.896 Acc@5 93.136
+# During Neural Architecture Search (NAS), we explore ResNet-34, ResNet-50, ResNet-53, and ResNet-56. After retraining with SPG algorithm, we retain only ResNet-50 and discard the others.
 torchrun --nproc_per_node=4 train.py\
     --data-path /home/cs/Documents/datasets/imagenet\
     --model resnet50 --output-dir resnet50 --weights ResNet50_Weights.IMAGENET1K_V1\
@@ -344,7 +278,7 @@ torchrun --nproc_per_node=4 train.py\
 
 To evaluate our models on ImageNet, run:
 
-```eval
+```bash
 
 cd image-classification
 
@@ -372,9 +306,9 @@ torchrun --nproc_per_node=4 train.py\
 
 To evaluate our models on COCO, run:
 
-```eval
+```bash
 
-cd semantic-segmentation
+cd ./examples/semantic-segmentation
 
 # eval baselines
 torchrun --nproc_per_node=4 train.py\
@@ -425,9 +359,9 @@ To evaluate our models on GLUE, SquAD, and SUPERB, please re-run the `transfer l
 
 
 For Network Architecture Search,  please run the following command to evaluate our SPG-trained ResNet-18 model:
-```eval
+```bash
 
-cd neural-architecture-search
+cd ./examples/neural-architecture-search
 
 # Required: Download our ResNet-18 weights to /path/to/neural-architecture-search/resnet18
 torchrun --nproc_per_node=4 train.py\