
**task**: `image-classification`  
**Backend:** `sagemaker-training`  
**Backend args:** `{'instance_type': 'ml.g4dn.2xlarge', 'supported_instructions': None}`  
**Number of evaluation samples:** `All dataset`  

Fixed parameters:
* **model_name_or_path**: `nateraw/vit-base-beans`
* **dataset**:
    * **path**: `beans`
    * **eval_split**: `validation`
    * **data_keys**: `{'primary': 'image'}`
    * **ref_keys**: `['labels']`
    * **calibration_split**: `train`
* **quantization_approach**: `dynamic`
* **calibration**:
    * **method**: `minmax`
    * **num_calibration_samples**: `100`
* **framework**: `onnxruntime`
* **framework_args**:
    * **opset**: `11`
    * **optimization_level**: `1`
* **aware_training**: `False`

Benchmarked parameters:
* **operators_to_quantize**: `['Add']`,  `['Add', 'MatMul']`
* **node_exclusion**: `[]`,  `['layernorm', 'gelu', 'residual', 'gather', 'softmax']`
* **per_channel**: `False`,  `True`

# Evaluation
## Non-time metrics
| operators_to_quantize |                      node_exclusion                      | per_channel |     | accuracy (original) | accuracy (optimized) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-----------------: | :------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |        0.980        |        0.980         |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |        0.980        |        0.980         |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |        0.980        |        0.980         |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |        0.980        |        0.980         |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |        0.980        |        0.980         |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |        0.980        |        0.980         |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |        0.980        |        0.980         |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |        0.980        |        0.980         |

## Time metrics
Time benchmarks were run for 15 seconds per config.


Below, time metrics for batch size = 1, input length = 32.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           200.50            |            63.00             |  \|  |           5.00            |           15.93            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           198.19            |            72.65             |  \|  |           5.07            |           13.80            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           191.44            |            63.27             |  \|  |           5.27            |           15.87            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           154.84            |            72.51             |  \|  |           6.47            |           13.80            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           155.84            |            130.95            |  \|  |           6.47            |            7.67            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           201.76            |            131.25            |  \|  |           5.00            |            7.67            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           198.96            |            128.82            |  \|  |           5.07            |            7.80            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           163.76            |            129.62            |  \|  |           6.13            |            7.73            |


Below, time metrics for batch size = 1, input length = 64.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           162.75            |            67.18             |  \|  |           6.20            |           14.93            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           159.69            |            72.77             |  \|  |           6.33            |           13.80            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           183.10            |            64.02             |  \|  |           5.47            |           15.67            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           157.21            |            64.16             |  \|  |           6.40            |           15.60            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           155.32            |            130.74            |  \|  |           6.47            |            7.67            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           198.56            |            162.51            |  \|  |           5.07            |            6.20            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           186.58            |            163.38            |  \|  |           5.40            |            6.13            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           199.75            |            131.46            |  \|  |           5.07            |            7.67            |


Below, time metrics for batch size = 1, input length = 128.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           160.58            |            67.65             |  \|  |           6.27            |           14.80            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           158.60            |            72.53             |  \|  |           6.33            |           13.80            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           200.46            |            62.95             |  \|  |           5.00            |           15.93            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           195.39            |            72.28             |  \|  |           5.13            |           13.87            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           197.59            |            128.80            |  \|  |           5.07            |            7.80            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           156.24            |            162.63            |  \|  |           6.47            |            6.20            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           157.25            |            129.13            |  \|  |           6.40            |            7.80            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           176.08            |            161.79            |  \|  |           5.73            |            6.20            |


Below, time metrics for batch size = 4, input length = 32.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           503.83            |            219.62            |  \|  |           2.00            |            4.60            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           603.26            |            266.15            |  \|  |           1.67            |            3.80            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           654.79            |            217.45            |  \|  |           1.53            |            4.60            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           654.33            |            219.54            |  \|  |           1.53            |            4.60            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           654.20            |            481.61            |  \|  |           1.53            |            2.13            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           609.81            |            632.73            |  \|  |           1.67            |            1.60            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           588.86            |            602.91            |  \|  |           1.73            |            1.67            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           666.98            |            655.32            |  \|  |           1.53            |            1.53            |


Below, time metrics for batch size = 4, input length = 64.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           656.87            |            216.32            |  \|  |           1.53            |            4.67            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           507.24            |            265.62            |  \|  |           2.00            |            3.80            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           655.36            |            219.61            |  \|  |           1.53            |            4.60            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           613.28            |            220.96            |  \|  |           1.67            |            4.53            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           656.30            |            652.72            |  \|  |           1.53            |            1.53            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           521.09            |            472.90            |  \|  |           1.93            |            2.13            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           655.37            |            473.77            |  \|  |           1.53            |            2.13            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           653.62            |            468.82            |  \|  |           1.53            |            2.13            |


Below, time metrics for batch size = 4, input length = 128.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           654.24            |            216.82            |  \|  |           1.53            |            4.67            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           657.16            |            240.11            |  \|  |           1.53            |            4.20            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           504.14            |            217.47            |  \|  |           2.00            |            4.60            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           655.94            |            220.12            |  \|  |           1.53            |            4.60            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           653.99            |            479.06            |  \|  |           1.53            |            2.13            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           642.48            |            666.28            |  \|  |           1.60            |            1.53            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           656.34            |            661.24            |  \|  |           1.53            |            1.53            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           661.86            |            472.49            |  \|  |           1.53            |            2.13            |


Below, time metrics for batch size = 8, input length = 32.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           1294.07           |            472.54            |  \|  |           0.80            |            2.13            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           1287.58           |            542.72            |  \|  |           0.80            |            1.87            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           1033.37           |            433.32            |  \|  |           1.00            |            2.33            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           1030.14           |            542.36            |  \|  |           1.00            |            1.87            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           953.27            |            926.14            |  \|  |           1.07            |            1.13            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           1173.01           |            995.22            |  \|  |           0.87            |            1.07            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           1280.07           |            926.97            |  \|  |           0.80            |            1.13            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           1283.70           |            927.87            |  \|  |           0.80            |            1.13            |


Below, time metrics for batch size = 8, input length = 64.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           1273.61           |            435.27            |  \|  |           0.80            |            2.33            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           1157.00           |            542.75            |  \|  |           0.87            |            1.87            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           968.85            |            537.65            |  \|  |           1.07            |            1.87            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           1107.66           |            472.53            |  \|  |           0.93            |            2.13            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           1270.30           |           1092.10            |  \|  |           0.80            |            0.93            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           1263.29           |           1012.66            |  \|  |           0.80            |            1.00            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           1007.19           |           1331.12            |  \|  |           1.07            |            0.80            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           1286.51           |           1317.96            |  \|  |           0.80            |            0.80            |


Below, time metrics for batch size = 8, input length = 128.

| operators_to_quantize |                      node_exclusion                      | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :------------------------------------------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           1188.98           |            537.58            |  \|  |           0.87            |            1.87            |
|  `['Add', 'MatMul']`  | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           951.31            |            489.40            |  \|  |           1.07            |            2.07            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `False`   |  \|  |           1278.73           |            537.52            |  \|  |           0.80            |            1.87            |
|  `['Add', 'MatMul']`  |                           `[]`                           |   `True`    |  \|  |           1005.38           |            440.01            |  \|  |           1.07            |            2.33            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `False`   |  \|  |           1265.55           |           1304.51            |  \|  |           0.80            |            0.80            |
|       `['Add']`       | `['layernorm', 'gelu', 'residual', 'gather', 'softmax']` |   `True`    |  \|  |           1186.54           |            934.09            |  \|  |           0.87            |            1.13            |
|       `['Add']`       |                           `[]`                           |   `False`   |  \|  |           1276.38           |           1319.84            |  \|  |           0.80            |            0.80            |
|       `['Add']`       |                           `[]`                           |   `True`    |  \|  |           981.81            |            940.69            |  \|  |           1.07            |            1.07            |


{"tags": ["vit"], "datasets": ["beans"], "metrics": ["accuracy"], "pipeline_tag": "image-classification"}
tensorboard vit image-classification dataset:beans region:us
TAGS
#tensorboard #vit #image-classification #dataset-beans #region-us 

task: 'image-classification'  

Backend: 'sagemaker-training'  

Backend args: '{'instance\_type': 'ml.g4dn.2xlarge', 'supported\_instructions': None}'  

Number of evaluation samples: 'All dataset'


Fixed parameters:


* model\_name\_or\_path: 'nateraw/vit-base-beans'
* dataset:
	+ path: 'beans'
	+ eval\_split: 'validation'
	+ data\_keys: '{'primary': 'image'}'
	+ ref\_keys: '['labels']'
	+ calibration\_split: 'train'
* quantization\_approach: 'dynamic'
* calibration:
	+ method: 'minmax'
	+ num\_calibration\_samples: '100'
* framework: 'onnxruntime'
* framework\_args:
	+ opset: '11'
	+ optimization\_level: '1'
* aware\_training: 'False'


Benchmarked parameters:


* operators\_to\_quantize: '['Add']', '['Add', 'MatMul']'
* node\_exclusion: '[]', '['layernorm', 'gelu', 'residual', 'gather', 'softmax']'
* per\_channel: 'False', 'True'


Evaluation
==========


Non-time metrics
----------------



Time metrics
------------


Time benchmarks were run for 15 seconds per config.


Below, time metrics for batch size = 1, input length = 32.



Below, time metrics for batch size = 1, input length = 64.



Below, time metrics for batch size = 1, input length = 128.



Below, time metrics for batch size = 4, input length = 32.



Below, time metrics for batch size = 4, input length = 64.



Below, time metrics for batch size = 4, input length = 128.



Below, time metrics for batch size = 8, input length = 32.



Below, time metrics for batch size = 8, input length = 64.



Below, time metrics for batch size = 8, input length = 128.




TAGS
#tensorboard #vit #image-classification #dataset-beans #region-us 

passage: TAGS
#tensorboard #vit #image-classification #dataset-beans #region-us 
