
**task**: `image-classification`  
**Backend:** `sagemaker-training`  
**Backend args:** `{'instance_type': 'ml.g4dn.2xlarge', 'supported_instructions': None}`  
**Number of evaluation samples:** `All dataset`  

Fixed parameters:
* **model_name_or_path**: `nateraw/vit-base-beans`
* **dataset**:
    * **path**: `beans`
    * **eval_split**: `validation`
    * **data_keys**: `{'primary': 'image'}`
    * **ref_keys**: `['labels']`
* **quantization_approach**: `dynamic`
* **node_exclusion**: `[]`
* **framework**: `onnxruntime`
* **framework_args**:
    * **opset**: `11`
    * **optimization_level**: `1`
* **aware_training**: `False`

Benchmarked parameters:
* **operators_to_quantize**: `['Add', 'MatMul']`,  `['Add']`,  `[]`
* **per_channel**: `False`,  `True`

# Evaluation
## Non-time metrics
| operators_to_quantize | per_channel |     | accuracy (original) | accuracy (optimized) |
| :-------------------: | :---------: | :-: | :-----------------: | :------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |        0.980        |        0.980         |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |        0.980        |        0.980         |
|       `['Add']`       |   `False`   |  \|  |        0.980        |        0.980         |
|       `['Add']`       |   `True`    |  \|  |        0.980        |        0.980         |
|         `[]`          |   `False`   |  \|  |        0.980        |        0.980         |
|         `[]`          |   `True`    |  \|  |        0.980        |        0.980         |

## Time metrics
Time benchmarks were run for 15 seconds per config.


Below, time metrics for batch size = 1, input length = 32.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           201.25            |            70.30             |  \|  |           5.00            |           14.27            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           203.52            |            72.48             |  \|  |           4.93            |           13.80            |
|       `['Add']`       |   `False`   |  \|  |           166.03            |            150.93            |  \|  |           6.07            |            6.67            |
|       `['Add']`       |   `True`    |  \|  |           200.82            |            163.17            |  \|  |           5.00            |            6.13            |
|         `[]`          |   `False`   |  \|  |           190.99            |            162.06            |  \|  |           5.27            |            6.20            |
|         `[]`          |   `True`    |  \|  |           155.15            |            162.52            |  \|  |           6.47            |            6.20            |


Below, time metrics for batch size = 1, input length = 64.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           165.85            |            70.60             |  \|  |           6.07            |           14.20            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           161.41            |            72.71             |  \|  |           6.20            |           13.80            |
|       `['Add']`       |   `False`   |  \|  |           200.45            |            129.40            |  \|  |           5.00            |            7.73            |
|       `['Add']`       |   `True`    |  \|  |           154.68            |            136.42            |  \|  |           6.47            |            7.40            |
|         `[]`          |   `False`   |  \|  |           166.97            |            162.15            |  \|  |           6.00            |            6.20            |
|         `[]`          |   `True`    |  \|  |           166.32            |            162.81            |  \|  |           6.07            |            6.20            |


Below, time metrics for batch size = 1, input length = 128.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           199.48            |            70.98             |  \|  |           5.07            |           14.13            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           199.65            |            71.78             |  \|  |           5.07            |           13.93            |
|       `['Add']`       |   `False`   |  \|  |           199.08            |            137.97            |  \|  |           5.07            |            7.27            |
|       `['Add']`       |   `True`    |  \|  |           189.93            |            162.45            |  \|  |           5.33            |            6.20            |
|         `[]`          |   `False`   |  \|  |           191.63            |            162.54            |  \|  |           5.27            |            6.20            |
|         `[]`          |   `True`    |  \|  |           200.38            |            162.55            |  \|  |           5.00            |            6.20            |


Below, time metrics for batch size = 4, input length = 32.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           655.84            |            243.33            |  \|  |           1.53            |            4.13            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           661.27            |            221.16            |  \|  |           1.53            |            4.53            |
|       `['Add']`       |   `False`   |  \|  |           662.84            |            529.28            |  \|  |           1.53            |            1.93            |
|       `['Add']`       |   `True`    |  \|  |           512.47            |            470.66            |  \|  |           2.00            |            2.13            |
|         `[]`          |   `False`   |  \|  |           562.81            |            501.77            |  \|  |           1.80            |            2.00            |
|         `[]`          |   `True`    |  \|  |           505.81            |            521.20            |  \|  |           2.00            |            1.93            |


Below, time metrics for batch size = 4, input length = 64.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           654.58            |            258.54            |  \|  |           1.53            |            3.93            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           617.44            |            234.05            |  \|  |           1.67            |            4.33            |
|       `['Add']`       |   `False`   |  \|  |           661.51            |            478.81            |  \|  |           1.53            |            2.13            |
|       `['Add']`       |   `True`    |  \|  |           657.01            |            660.23            |  \|  |           1.53            |            1.53            |
|         `[]`          |   `False`   |  \|  |           661.64            |            474.28            |  \|  |           1.53            |            2.13            |
|         `[]`          |   `True`    |  \|  |           661.29            |            471.09            |  \|  |           1.53            |            2.13            |


Below, time metrics for batch size = 4, input length = 128.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           654.80            |            219.38            |  \|  |           1.53            |            4.60            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           663.50            |            222.37            |  \|  |           1.53            |            4.53            |
|       `['Add']`       |   `False`   |  \|  |           625.56            |            529.02            |  \|  |           1.60            |            1.93            |
|       `['Add']`       |   `True`    |  \|  |           655.08            |            499.41            |  \|  |           1.53            |            2.07            |
|         `[]`          |   `False`   |  \|  |           655.92            |            473.01            |  \|  |           1.53            |            2.13            |
|         `[]`          |   `True`    |  \|  |           505.54            |            659.92            |  \|  |           2.00            |            1.53            |


Below, time metrics for batch size = 8, input length = 32.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           968.83            |            443.80            |  \|  |           1.07            |            2.27            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           1255.70           |            489.55            |  \|  |           0.80            |            2.07            |
|       `['Add']`       |   `False`   |  \|  |           1301.35           |            938.14            |  \|  |           0.80            |            1.07            |
|       `['Add']`       |   `True`    |  \|  |           1279.54           |            931.91            |  \|  |           0.80            |            1.13            |
|         `[]`          |   `False`   |  \|  |           1292.66           |           1318.07            |  \|  |           0.80            |            0.80            |
|         `[]`          |   `True`    |  \|  |           1290.35           |           1314.74            |  \|  |           0.80            |            0.80            |


Below, time metrics for batch size = 8, input length = 64.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           1305.45           |            438.06            |  \|  |           0.80            |            2.33            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           1296.68           |            450.40            |  \|  |           0.80            |            2.27            |
|       `['Add']`       |   `False`   |  \|  |           968.21            |            949.81            |  \|  |           1.07            |            1.07            |
|       `['Add']`       |   `True`    |  \|  |           1012.35           |           1317.46            |  \|  |           1.00            |            0.80            |
|         `[]`          |   `False`   |  \|  |           1213.91           |            961.79            |  \|  |           0.87            |            1.07            |
|         `[]`          |   `True`    |  \|  |           956.39            |            945.41            |  \|  |           1.07            |            1.07            |


Below, time metrics for batch size = 8, input length = 128.

| operators_to_quantize | per_channel |     | latency_mean (original, ms) | latency_mean (optimized, ms) |     | throughput (original, /s) | throughput (optimized, /s) |
| :-------------------: | :---------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: |
|  `['Add', 'MatMul']`  |   `False`   |  \|  |           1120.12           |            497.17            |  \|  |           0.93            |            2.07            |
|  `['Add', 'MatMul']`  |   `True`    |  \|  |           1289.50           |            443.46            |  \|  |           0.80            |            2.27            |
|       `['Add']`       |   `False`   |  \|  |           1294.65           |            930.97            |  \|  |           0.80            |            1.13            |
|       `['Add']`       |   `True`    |  \|  |           1181.21           |            933.82            |  \|  |           0.87            |            1.13            |
|         `[]`          |   `False`   |  \|  |           1245.61           |           1318.07            |  \|  |           0.87            |            0.80            |
|         `[]`          |   `True`    |  \|  |           1285.81           |           1318.82            |  \|  |           0.80            |            0.80            |


{"tags": ["vit"], "datasets": ["beans"], "metrics": ["accuracy"], "pipeline_tag": "image-classification"}
tensorboard vit image-classification dataset:beans region:us
TAGS
#tensorboard #vit #image-classification #dataset-beans #region-us 

task: 'image-classification'  

Backend: 'sagemaker-training'  

Backend args: '{'instance\_type': 'ml.g4dn.2xlarge', 'supported\_instructions': None}'  

Number of evaluation samples: 'All dataset'


Fixed parameters:


* model\_name\_or\_path: 'nateraw/vit-base-beans'
* dataset:
	+ path: 'beans'
	+ eval\_split: 'validation'
	+ data\_keys: '{'primary': 'image'}'
	+ ref\_keys: '['labels']'
* quantization\_approach: 'dynamic'
* node\_exclusion: '[]'
* framework: 'onnxruntime'
* framework\_args:
	+ opset: '11'
	+ optimization\_level: '1'
* aware\_training: 'False'


Benchmarked parameters:


* operators\_to\_quantize: '['Add', 'MatMul']', '['Add']', '[]'
* per\_channel: 'False', 'True'


Evaluation
==========


Non-time metrics
----------------



Time metrics
------------


Time benchmarks were run for 15 seconds per config.


Below, time metrics for batch size = 1, input length = 32.



Below, time metrics for batch size = 1, input length = 64.



Below, time metrics for batch size = 1, input length = 128.



Below, time metrics for batch size = 4, input length = 32.



Below, time metrics for batch size = 4, input length = 64.



Below, time metrics for batch size = 4, input length = 128.



Below, time metrics for batch size = 8, input length = 32.



Below, time metrics for batch size = 8, input length = 64.



Below, time metrics for batch size = 8, input length = 128.




TAGS
#tensorboard #vit #image-classification #dataset-beans #region-us 

passage: TAGS
#tensorboard #vit #image-classification #dataset-beans #region-us 
