README.md

---
pretty_name: "ComBack"
language: 
  - code
tags:
  - C++/C Code
  - Compiler Backend
license: "cc-by-4.0"
---


# ComBack: A Versatile Dataset for Enhancing Compiler Backend Development Efficiency

ComBack is a large-scale multi-platform compiler backend code dataset.
This repository contains all fine-tuned models and scripts for reproducing experimental results.


## Dataset Information

Details can be found at https://huggingface.co/datasets/docz-ict/ComBack

## Task Example

  - Statement-Level Completion: complete current statement.
  ```c++
  //Inputs:
  ...
  adjustReg(MBB,LastFrameDestroy, DL, SPReg, FPReg, -StackSize+RVFI->getVarArgsSaveSize() 
  //Ground Truth:
  MachineInstr::FrameDestroy);
  ```

  - Next-Statement Suggestion: predict the next statement.

   ```c++
  //Inputs:
  ...
  maxCallFrameSize = (maxCallFrameSize + AlignMask) & ~AlignMask;
  //Ground Truth:
  MFI -> setMaxCallFrameSize(maxCallFrameSize);
  ```


  - Code Generation: generate a function with function description in natrual language.

   ```c++
  //Inputs:
  getPointerRegClass: Returns a TargetRegisterClass used for pointer values.
  Target-Specific Value: Sparc, SP::I64RegsRegClass, SP::IntRegsRegClass.
  //Ground Truth:
  TargetRegisterClass *SparcRegisterInfo::getPointerRegClass(MachineFunction &MF ,unsigned Kind) {
      return Subtarget.is64Bit() ? &SP::I64RegsRegClass : &SP::IntRegsRegClass;
  }
  ```



## 1. Dependency

- python version == 3.8.1
- pip install -r requirements.txt

## 2. Fine-Tuning
We fine-tuned six pre-trained code language models on 8 Tesla V100 each with 16GB.
You can fine-tune each model on our datasets by running:

```shell
# Model Type Options: CodeBert, GraphCodeBert, UnixCoder, CodeT5, NatGen, CodeT5+
# Task Options: code-generation, code-completion, new-target-completion(Only for CodeT5+), new-target-generation(Only for CodeT5+)
bash ./Script/Model/{Model Type}/{Task}/run_fine_tuning*.sh
```



## 3. Reproducing Results in Table.2 

### Dataset split scheme
Split data of all 178 backends into train/valid/test set in the ratio of 80%:10%:10%

  - Dataset Info
  
  | Task | Train | Valid | Test |
  | ---- | ---- | ---- | ---- |
  | Statement-Level Comp. | 128,899(11.36M Token) | 16,112(1.43M Token)  | 16,113(1.43M Token) |
  | Next-Statement Sugg. | 173,052(15.69M Token) | 21,631(1.99M Token)   | 21,632(1.98M Token) |
  | Code Generation. | 36,236(5.10M Token) | 4,530(0.64M Token)  | 4,530(0.64M Token) |

### Reproducing results in Table.2 by running:

```shell
# Model Type Options: CodeBert, GraphCodeBert, UnixCoder, CodeT5, NatGen, CodeT5+
# Task Options: code-generation, code-completion
bash ./Script/Model/{Model Type}/{Task}/run_test.sh
```
### Results

  - Without Fine-Tuning
    |               | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. |
    |-------------|:-----------------:|:-----------------:|:----------------:|:----------------:|:----------:|:----------:|
    |    **Model**    |         EM        |         ED        |        EM        |        ED        |    BLEU4   |     ED     |
    | CodeBert-c      |        0.00       |        0.97       |       0.00       |       1.31       |     0.00    |     0.44    |
    | GraphCodeBert-c |        0.00       |        0.35       |       0.00       |       0.54       |     0.00    |     2.41    |
    | UnixCoder-base-nine     |        0.07       |       27.56       |        15.93       |        29.11       |    0.00    |    31.81   |
    | CodeT5-base        |        0.65       |       21.45       |        7.23       |        23.50       |     0.00    |     13.57    |
    | NatGen        |        0.00       |       13.52       |       0.02       |       15.95      |     0.01    |     28.76    |
    | CodeT5+-220m       |        0.02       |        7.24       |       0.12       |       9.87       |    0.00    |    12.33   |

  - Fine-Tuned
    |               | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. |
    |-------------|:-----------------:|:-----------------:|:----------------:|:----------------:|:----------:|:----------:|
    | **Model**         |         EM        |         ED        |        EM        |        ED        |    BLEU4   |     ED     |
    | CodeBert-c      |       53.84       |       77.44       |       52.67      |       70.82      |     23.54    |     54.63    |
    | GraphCodeBert-c |       43.00       |       71.89       |       47.10      |       61.31      |     20.73    |     48.83    |
    | UnixCoder-base-nine     |     **67.84**     |     **85.06**     |        58.51       |        75.31       |    56.24   |    73.45   |
    | CodeT5-base        |       66.38       |       84.34       |        58.52       |        76.03       |     70.87    |     80.45    |
    | NatGen        |       67.47       |       84.83       |     **60.30**    |     **76.84**    |     71.73    |     81.39    |
    | CodeT5+-220m       |       66.93       |       84.45       |       59.57      |       76.41      |  **75.28** |  **82.95** |




## 4. Reproducing Results in Table.3 





### Dataset split scheme

Take data of RISC-V,ARC,NVPTX both in GCC and LLVM as test set, split train/valid set in the ratio of 85%:15% of other CPU, MPU and GPU targets excluding RI5CY(RI5CY is custmoized based on RISCV)


  - Datset Info


    | Task | Train | Valid | Test |
    | ---- | ---- | ---- | ---- |
    | Statement-Level Comp. | 114,016(10.20M Token) | 20,121(1.81M Token)  | 6,645(0.58M Token) |
    | Next-Statement Sugg. | 152,114(14.10M Token) | 26,844(2.49M Token)   | 9,313(0.83M Token) |
    | Code Generation. | 30,633(4.44M Token) | 5,406(0.79M Token)  | 2,819(0.37M Token) |



### Input examples for ChatGPT-3.5-Turbo and Code-LLaMA-34B-Instruct
**Statement-Level Completion**
```cpp
//Prompt: Complete the last statement of this code snippet:
...
adjustReg(MBB,LastFrameDestroy, DL, SPReg, FPReg, -StackSize+RVFI->getVarArgsSaveSize() 
```
**Next-Statement Suggestion**
```cpp
//Prompt: Predict the next statement of this code snippet:
...
maxCallFrameSize = (maxCallFrameSize + AlignMask) & ~AlignMask;
```
**Code Generation**
```cpp
//Prompt: Create a function named "getPointerRegClass" for "Sparc" backend of LLVM Compiler. 
//The description of this function is "Returns a TargetRegisterClass used for pointer values". 
//It contains "Sparc", "SP::I64RegsRegClass", "SP::IntRegsRegClass" as target specific values.
```


### Reproducing results in Table.3 by running:
```shell
# Task Options: new-target-completion, new-target-generation
bash ./Script/Model/CodeT5+/{Task}/run_test_existing_type.sh

# ChatGPT
bash ./Script/Exp_Script/ChatGPT/run_chatgpt.sh

# Code-LLaMA
bash ./Script/Exp_Script/ChatGPT/run_codellama.sh
```



### Results


  - GCC

  |            | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. |
  |----------|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
  |            |    RISC-V   |    RISC-V   |     ARC     |     ARC     |    NVPTX    |    NVPTX    |    RISC-V   |    RISC-V   |     ARC     |     ARC     |    NVPTX    |    NVPTX    |   RISC-V   |   RISC-V   |     ARC    |     ARC    |    NVPTX   |    NVPTX   |
  | Model      |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |     BLEU4     |     ED     |     BLEU4     |     ED     |     BLEU4     |     ED     |
  | ChatGPT-3.5-Turbo    |    10.34    |    38.41    |    15.35    |    42.94    |    12.01    |    41.47    |     6.44    |     12.9    |     9.75    |    20.79    |     7.97    |    17.79    |   1.37  | 24.12 | 1.67  | 28.26 | 1.57 | 26.97 |
  | Code-LLaMA-34B |     0.41    |    19.07    |     0.85    |    16.77    |     0.56    |    18.22    |     1.58    |    13.54    |     2.66    |    17.95    |     2.47    |    16.59    |    1.67  | 27.89  | 1.71  | 30.49  | 1.57 |  27.65 |
  | CodeT5+-220m    |    **51.16**    |    **75.32**    |    **52.45**    |    **74.57**    |    **50.56**    |    **75.52**    |     **49.11**     |     **67.84**     |     **38.26**     |     **59.21**     |     **38.33**     |     **56.31**     |     **32.56**    |     **58.67**    |     **19.94**    |     **50.27**    |     **25.47**    |     **52.60**    |


  - LLVM

  |            | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. |
  |----------|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
  |            |    RISC-V   |    RISC-V   |     ARC     |     ARC     |    NVPTX    |    NVPTX    |    RISC-V   |    RISC-V   |     ARC     |     ARC     |    NVPTX    |    NVPTX    |   RISC-V   |   RISC-V   |     ARC    |     ARC    |    NVPTX   |    NVPTX   |
  | Model      |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |      EM     |      ED     |     BLEU4     |     ED     |     BLEU4     |     ED     |     BLEU4     |     ED     |
  | ChatGPT-3.5-Turbo    |     12.08     |     41.39     |     16.77     |     42.02     |     14.73     |     43.72     |     9.80     |    21.86    |    10.81    |    20.66    |    11.39    |    22.82    |    1.23 | 25.12 | 1.30 | 27.19 | 1.43 | 25.45 |
  | Code-LLaMA-34B |     0.45     |    17.61    |     0.61    |    17.21    |     0.99    |    17.23    |     1.75    |    15.04    |     0.42    |    11.27    |     2.42    |    16.25    |   1.43 | 27.24 | 1.61 | 32.12 | 1.59 | 28.08  |
  | CodeT5+-220m    |     **62.68**     |     **82.02**     |     **71.34**     |     **85.98**     |     **64.45**     |     **81.53**     |     **48.71**     |     **68.95**     |     **58.68**     |     **74.57**     |     **47.81**     |     **65.5**     |     **50.34**    |     **72.98**    |     **55.38**    |     **74.41**    |     **44.33**    |     **66.36**    |





## 5. Reproducing Results in Figure.6 

### Reproducing results in Table.4 by running:
```shell
# Task Options: new-target-completion, new-target-generation
bash ./Script/Model/CodeT5+/{Task}/run_test_existing_type.sh

# Fork-Flow
bash ./Script/Exp_Script/ForkFlow/run_forkflow.sh
```


### Results


  - GCC

  |  | RISCV | RISCV | ARC | ARC | NVPTX | NVPTX |
  |--------------  |-------  |-------  |-------  |------  |-------  |-------  |
  | Method | BLEU4 | ED | BLEU4 | ED | BLEU4 | ED |
  | ForkFlow Avg | 3.48 | 5.79 | 1.77 | 3.73 | 4.7 | 3.81 |
  | ForkFlow Max | 28.77 | 34.8 | 4.94 | 8.85 | 4.7 | 3.81 |
  | CodeT5+ | 32.56 | 58.67 | 25.47 | 52.6 | 19.94 | 50.27 |


  - LLVM

  |  | RISCV | RISCV | ARC | ARC | NVPTX | NVPTX |
  |--------------  |-------  |-------  |-------  |-------  |-------  |-------  |
  | Method | BLEU4 | ED | BLEU4 | ED | BLEU4 | ED |
  | ForkFlow Avg | 12.45 | 22.18 | 19.98 | 33.43 | 15.06 | 28.73 |
  | ForkFlow Max | 27.32 | 46.47 | 41.8 | 60.62 | 18.81 | 39.04 |
  | CodeT5+ | 50.34 | 72.98 | 55.38 | 74.41 | 44.33 | 66.36 |















## 6. Reproducing Results in Table.4



### Dataset split scheme


Take data of ARC,NVPTX both in GCC and LLVM as test set, split train/valid set in the ratio of 85%:15% of CPU targets excluding RISC-V and RI5CY

  - Datset Info


    | Task | Train | Valid | Test |
    | ---- | ---- | ---- | ---- |
    | Statement-Level Comp. | 87,018(7.78M Token) | 15,357(1.37M Token)  | 2,764(0.26M Token) |
    | Next-Statement Sugg. | 113,684(10.65M Token) | 20,063(1.87M Token)   | 4,029(0.38M Token) |
    | Code Generation. | 21,184(3.14M Token) | 3,739(0.55M Token)  | 1,372(0.18M Token) |


### Reproducing results in Table.4 by running:
```shell
# Task Options: new-target-completion, new-target-generation
bash ./Script/Model/CodeT5+/{Task}/run_test_new_type.sh
```



### Results

  - GCC

    |  | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. |
    |:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:  |:----------:  |:----------:  |:----------:  |
    |  | ARC(MPU) | ARC(MPU) | NVPTX(GPU) | NVPTX(GPU) | ARC(MPU) | ARC(MPU) | NVPTX(GPU) | NVPTX(GPU) | ARC(MPU) | ARC(MPU) | NVPTX(GPU) | NVPTX(GPU) |
    | Dataset | EM | ED | EM | ED | EM | ED | EM | ED | BLEU4 | ED | BLEU4 | ED |
    | -w GPU and MPU | 52.45 | 74.57 | 50.56 | 75.52 | 38.26 | 59.21 | 38.33 | 56.31 | 19.94 | 50.27 | 25.47 | 52.6 |
    | -w/o GPU and MPU | 50.53| 74.09 | 46.37 | 72.45 | 37.22 | 58.21 | 38.33 | 56.83 | 19.29 | 49.12 | 22.46 | 50.33 |
    |  **Decrease**  |  **1.92**  |  **0.48**  |  **4.19**  |  **3.07**  |  **1.04**  |  **1.00**  |  **0.00**  |  **-0.52**  |  **0.65**  |  **1.15**  |  **3.01**  |  **3.37**  |

  - LLVM
    |  | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. | Code. Gen. | Code. Gen. |
    |------------------  |:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------:  |:----------:  |:----------:  |:----------:  |
    |  | ARC(MPU) | ARC(MPU) | NVPTX(GPU) | NVPTX(GPU) | ARC(MPU) | ARC(MPU) | NVPTX(GPU) | NVPTX(GPU) | ARC(MPU) | ARC(MPU) | NVPTX(GPU) | NVPTX(GPU) |
    | Dataset | EM | ED | EM | ED | EM | ED | EM | ED | BLEU4 | ED | BLEU4 | ED |
    | -w GPU and MPU | 71.34 | 85.98 | 64.45 | 81.53 | 58.68 | 74.57 | 47.81 | 65.50 | 55.38 | 74.41 | 44.33 | 66.36 |
    | -w/o GPU and MPU | 69.82 | 85.59 | 60.04 | 79.85 | 58.26 | 73.75 | 46.28 | 63.92 | 49.62 | 70.26 | 42.94 | 65.43 |
    |  **Decrease**  |  **1.52**  |  **0.39**  |  **4.41**  |  **1.68**  |  **0.42**  |  **0.82**  |  **1.53**  |  **1.58**  |  **5.76**  |  **4.15**  |  **1.39**  |  **0.93**  |



## 7. Reproducing Results in Table.5



### Dataset split scheme


Take data of RI5CY in LLVM as test set, split train/valid set in the ratio of 85%:15% of CPU targets excluding RISC-V and including RISC-V

  - Datset Info
    - Excluding RISC-V

    | Task | Train | Valid | Test |
    | ---- | ---- | ---- | ---- |
    | Statement-Level Comp. | 87,018(7.78M Token) | 15,357(1.37M Token)  | 721(0.04M Token) |
    | Next-Statement Sugg. | 113,684(10.65M Token) | 20,063(1.87M Token)   | 1,035(0.06M Token) |
    | Code Generation. | 21,184(3.14M Token) | 3,739(0.55M Token)  | 219(0.02M Token) |

    - Including RISC-V
  
    | Task | Train | Valid | Test |
    | ---- | ---- | ---- | ---- |
    | Statement-Level Comp. | 90,316(8.06M Token) | 15,940(1.42M Token)  | 721(0.04M Token) |
    | Next-Statement Sugg. | 118,175(11.04M Token) | 20,856(1.94M Token)   | 1,035(0.06M Token) |
    | Code Generation. | 22,413(3.30M Token) | 3,957(0.58M Token)  | 219(0.02M Token) |



### Reproducing results in Table.5 by running:
```shell
# Task Options: new-target-completion, new-target-generation
bash ./Script/Model/CodeT5+/{Task}/run_test_itr_exp.sh
```



### Results

  | | Stmt. Comp. | Stmt. Comp. | Next. Sugg. | Next. Sugg. | Code. Gen. | Code. Gen. |
  |:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:----------: |:----------: |
  | Dataset | EM | ED | EM | ED | BLEU4 | ED |
  | -w/o RISC-V | 66.16 | 83.79 | 57.29 | 74.73 | 54.41 | 75.41 |
  | -w RISC-V | 74.06 | 87.91 | 67.25 | 81.28 | 79.46 | 89.92 |
  | **Diff** | **7.90** | **4.12** | **9.96** | **6.55** | **25.05** | **14.51** |





## Citation
```
@inproceedings{zhong2024comback,
  title={ComBack: A Versatile Dataset for Enhancing Compiler Backend Development Efficiency},
  author={Ming Zhong, Fang Lyu, Lulin Wang, Hongna Geng, Lei Qiu, Huimin Cui, Xiaobing Feng},
  booktitle={Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2024}
}
```